Structured visual positioning method, system and equipment based on intention recognition

Abstract

The invention relates to the technical field of visual positioning, and discloses a structured visual positioning method, system and device based on intention recognition, and the method comprises the steps: obtaining a to-be-positioned image and a current voice instruction of a user; inputting the current voice instruction into a pre-trained intention recognition model to perform semantic structuring processing so as to construct a semantic topological graph; wherein the semantic topological graph is used for describing target structured data corresponding to the current voice instruction; reasoning based on the semantic topological graph to obtain a semantic reference corresponding to the current voice instruction; and performing feature alignment on the semantic reference and the to-be-positioned image so as to position a target object matched with the semantic reference in the to-be-positioned image. According to the invention, through semantic structuring processing on the voice instruction, accurate reasoning can be carried out on the semantics containing the spoken language instruction, so that the robustness of a visual positioning system is greatly improved, and the visual positioning effect in a daily dialogue scene is improved.

Description

Technical Field

[0001] The present application relates to the field of visual positioning technology, and in particular to a structured visual positioning method, system and device based on intent recognition. Background Art

[0002] Visual Grounding is a technology that matches natural language descriptions with specific visual content in an image, and is widely used in many fields such as human-computer interaction, augmented reality, robot navigation, and medical image analysis. Its input usually includes an image and a language description of an object or scene. The image and language description are multimodally fused to output the bounding box or segmentation of the corresponding object or scene in the image.

[0003] In related technologies, visual positioning systems usually have high requirements for the completeness of the input language description. Users usually need to accurately describe their needs in written language to obtain relatively accurate visual positioning results. However, the language descriptions in daily conversation scenes generally contain too much noise information, which will seriously affect the accuracy of the visual positioning results, resulting in poor system robustness, which in turn limits the application of visual positioning. Summary of the invention

[0004] The present application provides a structured visual positioning method, system and device based on intent recognition, which solves the technical problem that the current visual positioning system has poor robustness when applied to daily conversation scenarios, resulting in a decrease in the accuracy of visual positioning. The present application can accurately infer the semantics of spoken language instructions through semantically structured processing of voice commands, thereby greatly improving the robustness of the visual positioning system and facilitating the improvement of the visual positioning effect in daily conversation scenarios.

[0005] In order to achieve the above objectives, the main technical solutions adopted in this application include:

[0006] In a first aspect, an embodiment of the present application provides a structured visual positioning method based on intent recognition, characterized in that the method includes:

[0007] Obtain the image to be located and the user's current voice command;

[0008] Inputting the current voice command into a pre-trained intention recognition model for semantic structuring processing to construct a semantic topology map; wherein the semantic topology map is used to describe the target structured data corresponding to the current voice command;

[0009] Performing reasoning based on the semantic topology graph to obtain the semantic reference corresponding to the current voice instruction;

[0010] Feature alignment is performed on the semantic reference and the image to be located, so as to locate a target object matching the semantic reference in the image to be located.

[0011] The structured visual positioning method proposed in the embodiment of the present application performs deep semantic analysis and structural processing on the voice commands issued by the user through the intention recognition model, thereby constructing a semantic topological map that can accurately describe the user's intention, and obtains semantic references through logical reasoning of the semantic topological map to achieve accurate and rapid understanding of the user's semantic intentions. Finally, the obtained semantic references are aligned with the visual features in the image through multimodal features, so as to accurately locate the target object corresponding to the semantics of the user's command in the image to be located. Therefore, the embodiment of the present application can effectively and accurately parse voice commands containing a lot of colloquial noise, ensure that the user's true intentions can be efficiently and accurately identified in a complex language environment, thereby making the multimodal fusion in the visual positioning process more stable and efficient, greatly improving the semantic understanding ability and robustness, and effectively improving the user experience.

[0012] Optionally, in some embodiments of the present application, the inputting the current voice instruction into a pre-trained intention recognition model for semantic structuring processing to construct a semantic topology graph includes:

[0013] Performing semantic analysis on the current voice instruction by using the intention recognition model to obtain a current semantic factor in the current voice instruction;

[0014] Outputting according to a preset data format based on the current semantic factor to obtain the semantic topology map;

[0015] The semantic factors include the task, target, relation triples and attributes corresponding to the current voice instruction.

[0016] The embodiment of the present application utilizes an intent recognition model to perform semantic analysis on the user's spoken voice commands, and identifies the current semantic factors representing the user's intent, including tasks, goals, relationship triples, and attributes, etc. By extracting and organically combining the semantic factors, the originally ambiguous, redundant, and spoken voice commands are reconstructed into a semantic topology map with clear semantic logic. Therefore, the information interference caused by spoken noise can be effectively reduced, thereby significantly improving the fault tolerance and semantic reasoning level for complex language expressions, which is beneficial to subsequent multimodal fusion and feature alignment, thereby improving the accuracy of visual positioning.

[0017] Optionally, in some embodiments of the present application, the training process of the intent recognition model includes:

[0018] Determining a preset number of first corpora in a specialized corpus of visual positioning descriptions; wherein the specialized corpus is constructed based on historical voice instructions;

[0019] In response to the labeling operation on the preset number of first corpora, a labeling sample is obtained;

[0020] Based on the annotated samples, automatically annotate the second corpora other than the first corpus in the specialized corpus, and filter and screen the automatically annotated second corpora to construct a candidate sample set using the filtered second corpora and the first corpus;

[0021] The candidate sample set is proportionally adjusted according to the labeled categories to obtain a training sample set, and the initial recognition model is trained using the training sample set to obtain the intent recognition model.

[0022] The embodiments of the present application greatly improve the efficiency of corpus annotation and reduce the difficulty of manual annotation through automatic annotation, improve the quality of the training sample set through subsequent filtering and screening, and ensure the data diversity and balance of the training sample set by proportionally adjusting the candidate sample set according to the annotation category, so as to avoid the annotation of certain categories being too concentrated or scarce and affecting the training effect of the initial recognition model, which is conducive to the intent recognition model to output an accurate semantic topology map.

[0023] Optionally, in some embodiments of the present application, the automatic annotation includes:

[0024] Determine the prompt words according to the requirements of the annotation task and output format;

[0025] The second corpus is annotated using the CoT technology according to the prompt word and the annotated sample.

[0026] Optionally, in some embodiments of the present application, before outputting according to a preset data format based on the current semantic factor to obtain the semantic topology map, the method further includes:

[0027] Using a sliding window method, determine the historical structured data corresponding to the occurrence time of the current voice command, select key semantic factors from the semantic factors of the historical structured data, and store the key semantic factors as context information;

[0028] The outputting according to a preset data format based on the current semantic factor to obtain the semantic topology map includes:

[0029] The semantic topology map is output according to the current semantic factor and the context information in a preset data format.

[0030] In the process of constructing a semantic topology map, the embodiment of the present application introduces context information to enhance the semantic understanding of the user's current voice command. Therefore, the present application intelligently supplements the current semantic factors through context information, so that it can effectively adapt to the situation where the current voice command has unclear reference, semantic disconnection or strong context dependence, thereby reducing the dependence on the user's complete description, making the semantic understanding more accurate and smooth, and greatly improving the user experience. In addition, the embodiment of the present application uses a sliding window to accurately select and store potentially relevant context information. The user can configure the sliding window according to their own needs to dynamically adjust the selection range of historical structured data, thereby improving the flexibility and configurability of context information management.

[0031] Optionally, in some embodiments of the present application, outputting the semantic topology map in a preset data format according to the current semantic factor and the context information includes:

[0032] Determining initial structured data corresponding to the current semantic factor;

[0033] The context information is matched with the initial structured data, and the initial structured data is contextually supplemented according to the matching result, so as to obtain the semantic topology map in the preset data format.

[0034] The embodiment of the present application realizes the structured expression of the current voice command through the semantic topology map, and can accurately and clearly express the information logic in the current voice command in combination with contextual information, thereby realizing the conversion from spoken expression to structured semantic data, and providing a reliable basis for subsequently determining the target object that the user actually intends to refer to through semantic reasoning, and accurately locating the target object in the image.

[0035] Optionally, in some embodiments of the present application, the reasoning based on the semantic topology graph to obtain the semantic reference corresponding to the current voice instruction includes:

[0036] Perform reasoning based on the target in the current voice instruction and the semantic topology graph to obtain attribute and relationship triples related to the target;

[0037] The semantic reference is generated according to the attribute and relationship triples related to the target.

[0038] The embodiment of the present application utilizes a semantic topological graph to clearly express the logical relationship of the current voice command, so as to realize the conversion of complex natural language into structured expression. Therefore, it is possible to quickly and accurately infer the attribute and relationship triples related to the target, and express the user's true intention with concise semantic representation, providing a clear and accurate information basis for the subsequent alignment of semantic features with visual features and the realization of target object positioning, thereby improving the ability to accurately identify user intentions in spoken voice commands.

[0039] Optionally, in some embodiments of the present application, the performing feature alignment on the semantic reference and the image to be located to locate a target object matching the semantic reference in the image to be located includes:

[0040] Extracting semantic features from the semantic reference and obtaining a semantic feature vector;

[0041] Extracting image features from the image to be located and obtaining an image feature vector;

[0042] The semantic feature vector is matched with the image feature vector, and the target object is determined according to the matching result.

[0043] In a second aspect, an embodiment of the present application provides a structured visual positioning system based on intent recognition, the system comprising:

[0044] An acquisition module is used to acquire the image to be located and the user's current voice command;

[0045] An intention recognition module, used for inputting the current voice instruction into a pre-trained intention recognition model for semantic structural processing to construct a semantic topological map; wherein the semantic topological map is used to describe the target structured data corresponding to the current voice instruction;

[0046] A reference reasoning module, used for performing reasoning based on the semantic topology graph to obtain the semantic reference corresponding to the current voice instruction;

[0047] The feature alignment module is used to perform feature alignment on the semantic reference and the image to be located, so as to locate a target object matching the semantic reference in the image to be located.

[0048] The structured visual positioning system proposed in the embodiment of the present application performs deep semantic analysis and structural processing on the voice commands issued by the user through the intention recognition model, thereby constructing a semantic topological map that can accurately describe the user's intention, and obtains semantic references through logical reasoning of the semantic topological map to achieve accurate and rapid understanding of the user's semantic intentions. Finally, the obtained semantic references are aligned with the visual features in the image through multimodal features, so as to accurately locate the target object corresponding to the semantics of the user's command in the image to be located. Therefore, the embodiment of the present application can effectively and accurately parse voice commands containing a lot of colloquial noise, ensure that the user's true intentions can be efficiently and accurately identified in a complex language environment, thereby making the multimodal fusion in the visual positioning process more stable and efficient, greatly improving the semantic understanding ability and robustness, and effectively improving the user experience.

[0049] In a third aspect, an embodiment of the present application provides a computer device, including:

[0050] A memory and a processor, wherein the memory and the processor are communicatively connected to each other, the memory stores computer instructions, and the processor executes the structured visual positioning method based on intent recognition described in the above embodiment by executing the computer instructions. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] In order to more clearly illustrate the specific implementation methods of the present application or the technical solutions in the prior art, the drawings required for use in the specific implementation methods or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some implementation methods of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0052] Figure 1 A flowchart of a structured visual positioning method based on intent recognition proposed in an embodiment of the present application;

[0053] Figure 2 A flowchart of semantic structuring processing proposed in one embodiment of the present application;

[0054] Figure 3 A schematic diagram of the structure of a semantic topology graph proposed in one embodiment of the present application;

[0055] Figure 4 A flowchart of semantic structuring processing proposed in another embodiment of the present application;

[0056] Figure 5 A schematic diagram of the structure of a semantic topology graph proposed in another embodiment of the present application;

[0057] Figure 6A schematic diagram of the structure of a structured visual positioning system based on intent recognition proposed in an embodiment of the present application;

[0058] Figure 7 A schematic diagram of the structure of a computer device proposed in an embodiment of the present application. DETAILED DESCRIPTION

[0059] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present application.

[0060] Visual Grounding is a technology that matches natural language descriptions with specific visual content in an image, and is widely used in many fields such as human-computer interaction, augmented reality, robot navigation, and medical image analysis. Its input usually includes an image and a language description of an object or scene. The image and language description are fused multimodally. The relevant semantic features of the language modality input are first extracted, and then fused with the information of the visual modality. Finally, the obtained features are used to locate the corresponding object or scene in the visual content, thereby outputting the bounding box or segmentation of the corresponding object or scene in the image.

[0061] In some application scenarios, related technologies often emphasize visual positioning based on well-formed semantics, and have high requirements for the completeness of the input language description. In the process of human-computer interaction, users usually need to accurately describe their needs in written language to obtain relatively accurate visual positioning results. Once the language used is slightly longer and has some changes or is in the context of daily spoken conversation, it often causes serious performance degradation and even causes confusion in semantic understanding. For example, when using a household robot, the user is in a home environment and usually speaks spoken commands such as "I'm a little hungry, wow! There are so many foods on the table, I want that apple pie, uh... no, it's cake", and the command samples input into the model training phase are usually accurate and concise sentences such as "the cake on the table". Therefore, when the spoken commands are full of noise, it is easy to cause the model performance to degrade or even misunderstanding. For example, the model may try to align the semantic features of words such as "hungry", "table", "apple pie", "uh" or "cake" with the visual features, but there is no area related to modal particles such as "uh" on the image, and "apple pie" is a wrong target. Therefore, the noise of the spoken commands will interfere with the model's understanding of the semantics of "cake" that the user really wants to express.

[0062] In addition, in some other application scenarios, related technologies cannot trace the user's historical conversation information. Users can only fully describe the object they want to refer to each time, and cannot connect it with previous references. This cannot meet the needs of human-computer interaction and reduces the intelligent performance of semantic recognition. For example, if the user has a human-computer conversation history including "find the black shirt on the sofa" and "find the red cup on the table", and the user says "find the clothes just now", the model will be confused about the command "find the clothes just now", make mistakes, or require the user to provide a complete expression, which seriously affects the efficiency of visual positioning and greatly reduces the user experience.

[0063] Therefore, the visual positioning system in the related art is difficult to adapt to daily conversation scenarios with too much noise, and its robustness is poor, which in turn limits the application of visual positioning.

[0064] Therefore, in daily conversation scenarios where there is too much noise information, the visual positioning results of related technologies have low accuracy and poor system robustness, which makes it difficult to adapt to the intention recognition needs of daily conversation scenarios, thereby limiting the application of visual positioning.

[0065] According to an embodiment of the present application, an embodiment of a structured visual positioning method based on intent recognition is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0066] In this embodiment, a structured visual positioning method based on intent recognition is provided, which can be used for the above-mentioned smart assistants, augmented reality, smart homes, robot navigation and other smart devices, such as mobile phones, tablet computers and control terminals, etc. Figure 1 is a flow chart of a structured visual positioning method according to an embodiment of the present application, such as Figure 1 As shown, the process includes the following steps:

[0067] Step S1, obtaining the image to be located and the user's current voice command.

[0068] In the embodiment of the present application, the image to be located includes but is not limited to the image of the scene where the user is located acquired by the image acquisition terminal of the camera and sensor, and the image input by the user. The user's current voice command includes but is not limited to the voice information acquired by the user through the voice acquisition terminal such as a microphone.

[0069] Step S3, input the current voice command into a pre-trained intent recognition model for semantic structuring processing to construct a semantic topology map; wherein the semantic topology map is used to describe the target structured data corresponding to the current voice command.

[0070] The embodiment of the present application inputs the current voice command into the intention recognition model. The task of the intention recognition model is to perform semantic understanding of the current voice command. In this process, a semantic topology map is constructed based on the semantic understanding of the current voice command. The semantic topology map can convert the current voice command with natural language characteristics into a structured expression. This structured expression can represent the logical relationship and information dependency in the current voice command in a clear hierarchical manner, so that the semantic reference corresponding to the current voice command can be accurately inferred based on the semantic topology map.

[0071] Step S5, performing reasoning based on the semantic topology graph to obtain the semantic reference corresponding to the current voice command.

[0072] The embodiment of the present application analyzes the topological relationship in the semantic topological map to infer the above-mentioned semantic reference. Through the semantic reference, the true intention of the user in issuing the current voice command can be concisely, clearly and accurately identified, thereby providing an accurate semantic basis for visual positioning.

[0073] Step S7, aligning features of the semantic reference and the image to be located, so as to locate the target object described by the current voice instruction in the image to be located.

[0074] The embodiment of the present application obtains accurate semantic reference through the above steps S1 to S5, and then determines the matching target object in the image to be located by aligning the features of the semantic reference and the image to be located, and displays the area where the target object is located in the form of a bounding box or segmentation.

[0075] The structured visual positioning method proposed in the embodiment of the present application performs deep semantic analysis and structural processing on the voice commands issued by the user through the intention recognition model, thereby constructing a semantic topological map that can accurately describe the user's intention, and obtains semantic references through logical reasoning of the semantic topological map to achieve accurate and rapid understanding of the user's semantic intentions. Finally, the obtained semantic references are aligned with the visual features in the image through multimodal features, so as to accurately locate the target object corresponding to the semantics of the user's command in the image to be located. Therefore, the embodiment of the present application can effectively and accurately parse voice commands containing a lot of colloquial noise, ensure that the user's true intentions can be efficiently and accurately identified in a complex language environment, thereby making the multimodal fusion in the visual positioning process more stable and efficient, greatly improving the semantic understanding ability and robustness, and effectively improving the user experience.

[0076] In some optional embodiments of the present application, the training process of the above-mentioned intention recognition model includes:

[0077] Determining a preset number of first corpora in a specialized corpus of visual location descriptions; wherein the specialized corpus is constructed based on historical speech instructions;

[0078] In response to the labeling operation on a preset number of first corpora, labeling samples are obtained;

[0079] Based on the annotated samples, the second corpus other than the first corpus in the special corpus is automatically annotated, and the automatically annotated second corpus is filtered and screened to construct a candidate sample set using the filtered second corpus and the first corpus;

[0080] The candidate sample set is proportionally adjusted according to the labeled categories to obtain a training sample set, and the initial recognition model is trained using the training sample set to obtain an intent recognition model.

[0081] Furthermore, the above automatic annotation includes:

[0082] Determine the prompt words according to the requirements of the annotation task and output format;

[0083] The second corpus is annotated using the CoT (Chain of Thought) technology according to the prompt words and annotation samples.

[0084] Specifically, in the training phase of the intent recognition model, we first collect dialogue segments related to visual positioning in real human-computer dialogues to build a special corpus of visual positioning descriptions, and use closed-source large models such as GPT, ChatGLM, and Claude to expand the special corpus. Then, a small amount of corpus from the special corpus is selected as the first corpus and manually annotated to provide annotation references for subsequent automatic annotation. For example, if the first corpus is "There is a wooden table, and there is a red cup on the table", the first corpus obtained through manual annotation is as follows: "Task: visual positioning; Target: Cup; Relation triple: The table supports the cup; Attribute 1: Red cup; Attribute 2: Wooden table".

[0085] After completing the manual annotation of the first corpus, the first corpus before and after the annotation is used as an annotation sample, and the second corpus other than the first corpus in the special corpus is automatically annotated based on the annotation sample.

[0086] It should be noted that the number of the first corpus is less than the number of the second corpus, that is, the embodiment of the present application selects a small amount of corpus from the special corpus as the first corpus and manually annotates it, thereby forming an annotation reference for the remaining second corpus.

[0087] Further, in some embodiments of the present application, in the annotation stage, specifically, prompt words are set to define tasks and output format requirements, wherein the task is defined as visual positioning, and the output format requirement is output in the format of the above-mentioned "task", "goal", "relationship triplet" and "attribute". After setting the prompt words, the second corpus is automatically annotated using CoT (Chain of Thought) technology. CoT technology is a method for problem solving or reasoning, and its core is to decompose complex problems into multiple simple logical steps, so as to analyze and solve the problems more clearly step by step according to the logical steps, so as to obtain the final answer. The embodiment of the present application uses CoT technology to realize the step-by-step thinking of the annotation of the above-mentioned second corpus according to the tasks defined in the prompt words, and outputs the annotated second corpus according to the output format requirements defined in the prompt words.

[0088] Therefore, the embodiment of the present application improves the efficiency of corpus annotation and minimizes manual intervention by using the automatic annotation realized by CoT technology. At the same time, according to different task requirements, the prompt words can be adjusted to flexibly set the annotation strategy to adapt to the diverse corpus annotation requirements.

[0089] After completing the annotation of all corpora in the special corpus, in order to ensure the accuracy of the annotation, a screening stage is also set in some embodiments of the present application. In the screening stage, specifically, the output annotated second corpus is compared with the output format requirements defined in the prompt word, the annotation results that do not meet the output format requirements are screened out, and the remaining annotated second corpus is placed in the candidate sample set. And because the first corpus adopts a manual annotation method, its annotation results must meet the output format requirements, so the annotated first corpus will inevitably be placed in the candidate sample set. In addition, in order to ensure the diversity and balance of the candidate sample set, the embodiment of the present application also adjusts the proportion according to the annotation category in the screening stage to obtain a training sample set, for example, according to the category of the "target" in the annotation result, the annotation type is determined, such as "target" is a description of the scene or a description of the object, etc., so the embodiment of the present application adjusts the proportion of the annotation category to avoid excessive concentration or scarcity of annotations of certain categories, so as not to affect the training effect of the intention recognition mode.

[0090] After obtaining the training sample set, the training sample set is input into the initial recognition model for training. The embodiment of the present application sets a corresponding prompt engineering strategy to optimize the performance of the large intent model in the visual positioning task. Specifically, firstly, a training instruction is set. The training instruction corresponds to the prompt word used in the above-mentioned corpus annotation process. The task to be completed by the intent recognition model is clearly defined as visual positioning through the training instruction, and the expected output format is clearly defined as: "Task:...; Target:...; Relation triple:...; Attribute 1:...;... Attribute n:...".

[0091] This output format not only improves the consistency of the data, but also facilitates subsequent feature alignment and task processing. After that, the set training instructions and the above training sample set are input into the initial recognition model for training, thereby obtaining an intent recognition model that meets the semantic understanding requirements.

[0092] Therefore, the embodiments of the present application greatly improve the efficiency of corpus annotation and reduce the difficulty of manual annotation through automatic annotation, improve the quality of the training sample set through subsequent filtering and screening, and ensure the data diversity and balance of the training sample set by proportionally adjusting the candidate sample set according to the annotation category, thereby avoiding the excessive concentration or scarcity of annotations of certain categories that affects the training effect of the initial recognition model, which is conducive to the intent recognition model to output an accurate semantic topology map.

[0093] In addition, during the training process of the intent recognition model in the embodiments of the present application, in order to further improve the accuracy of the intent recognition model, in some embodiments of the present application, the initial recognition model is also trained for context understanding, so that the intent recognition model obtained through training can better understand the intention referred to by the current voice based on the user's conversation history, thereby improving the semantic parsing capability of the intent recognition model.

[0094] Specifically, the embodiment of the present application obtains the first corpus and the second corresponding historical conversation in the training sample set, and forms annotated historical corpus with these historical conversations in the same way as the above-mentioned automatic annotation, that is, the historical conversations are annotated using the CoT technology according to the prompt words and the annotated samples, thereby obtaining a context sample set composed of the historical corpus. After that, the set training instructions, the above-mentioned training sample set and the above-mentioned context sample set are jointly input into the initial recognition model for training, thereby obtaining an intent recognition model that meets the semantic understanding requirements.

[0095] In some optional embodiments of the present application, such as Figure 2 As shown, the above step S3 includes the following steps:

[0096] Step S31, performing semantic analysis on the current voice command through the intention recognition model to obtain the current semantic factor in the current voice command;

[0097] Step S33, outputting according to a preset data format based on the current semantic factor to obtain a semantic topology map;

[0098] Among them, the semantic factors include the task, target, relation triples and attributes corresponding to the current voice command.

[0099] Specifically, the intention recognition model in the embodiment of the present application uses the open source large language model llama as a language understanding engine, and performs semantic analysis on the user's current voice command according to the prompt words pre-set in the training stage, wherein the prompt words trigger the intention recognition model to start semantic analysis of the current voice command by defining the task as visual positioning, so as to obtain the corresponding current semantic factors. For example, the current voice command input by the user to the consciousness recognition model is "Well... Can you help me take a look at that pot of flowers? It's the one that is about to wither. By the way, remember it is on the balcony? I should water or prune it, thank you!" The consciousness recognition model performs semantic analysis on the current voice command, thereby obtaining the target structured data composed of the current semantic factors: "Task: Visual positioning; Target: Flower; Relationship triple: Balcony supports flowers; Attribute: Flowers that are about to wither".

[0100] In some embodiments of the present application, the above-mentioned preset data format is the format of a semantic topology graph, which can transform the complexity of natural language into a clearly structured expression with a hierarchy to complete subsequent efficient visual-language interaction. In the semantic topology graph, the scene described by the user's current semantic instruction is expressed as a graph structure. The items in the current semantic instruction are the item nodes of the graph. The relationship between the item nodes is a triplet of the relationship between items. At the same time, items also have attributes, so the attribute nodes corresponding to the item nodes are set. Figure 3 As shown, for the current semantic factor in the above target structured data, the item node "flower" is generated corresponding to the current semantic factor "target", the item node "balcony" is set according to the current semantic factor "relationship triplet", and a line is connected between the item node "flower" and the item node "balcony", and the arrow direction of the line points to the item node "flower" to indicate the supporting relationship of "balcony" to "flower", that is, the graph structure expression of "relationship triplet: balcony supports flower" is realized, and the attribute node "withering" is set according to the current semantic factor "attribute".

[0101] The embodiment of the present application utilizes an intent recognition model to perform semantic analysis on the user's spoken voice commands, and identifies the current semantic factors representing the user's intent, including tasks, goals, relationship triples, and attributes, etc. By extracting and organically combining the semantic factors, the originally ambiguous, redundant, and spoken voice commands are reconstructed into a semantic topology map with clear semantic logic. Therefore, the information interference caused by spoken noise can be effectively reduced, thereby significantly improving the fault tolerance and semantic reasoning level for complex language expressions, which is beneficial to subsequent multimodal fusion and feature alignment, thereby improving the accuracy of visual positioning.

[0102] In some embodiments of the present application, the above step S5 includes:

[0103] Reasoning is performed based on the target and semantic topology graph in the current voice command to obtain attribute and relationship triples related to the target;

[0104] Generate semantic references based on attribute and relation triples related to the target.

[0105] For example, in Figure 3 On the basis of the semantic topology diagram shown, according to the relationship between each node and its connection, the content corresponding to the node is spliced ​​into a semantic reference of "the flowers on the balcony that are about to wither", thereby realizing accurate intention recognition of the user's current voice command.

[0106] The embodiment of the present application utilizes a semantic topological graph to clearly express the logical relationship of the current voice command, so as to realize the conversion of complex natural language into structured expression. Therefore, it is possible to quickly and accurately infer the attribute and relationship triples related to the target, and express the user's true intention with concise semantic representation, providing a clear and accurate information basis for the subsequent alignment of semantic features with visual features and the realization of target object positioning, thereby improving the ability to accurately identify user intentions in spoken voice commands.

[0107] In some embodiments of the present application, Figure 4 As shown, before the above step S33, the method further includes the following steps:

[0108] Step S32, using a sliding window method, determines the historical structured data corresponding to the time when the current voice command occurs, selects key semantic factors from the semantic factors of the historical structured data, and stores the key semantic factors as context information.

[0109] Therefore, the above step S33 also includes:

[0110] According to the current semantic factors and context information, the semantic topology map is output in a preset data format.

[0111] The embodiment of the present application uses a sliding window approach to manage contextual information. Since the intent recognition model can output structured semantic factors, the embodiment of the present application can concisely and accurately store effective information of historical structured data by using a small number of key semantic factors, thereby replacing the long text input of the user's historical voice commands, saving storage space for contextual information, and facilitating improving the historical tracing scope of the current voice command.

[0112] Furthermore, in some embodiments of the present application, outputting a semantic topology map in a preset data format according to current semantic factors and context information includes:

[0113] Determine the initial structured data corresponding to the current semantic factor;

[0114] The context information is matched with the initial structured data, and the initial structured data is supplemented with context according to the matching result to obtain a semantic topology map in a preset data format.

[0115] Specifically, users can configure the size of the sliding window. For example, the sliding window can store 4096 words, and the default setting is to save the key semantic factors of 5 historical structured data. The movement of the sliding window will be triggered in each round of human-computer dialogue to ensure the continuity of the context. In special cases (such as topic switching), the sliding window can be reset.

[0116] In some embodiments of the present application, the above-mentioned key semantic factors are “target” and “semantic reference” in the semantic factors of historical structured data.

[0117] For example, the user's current voice command is "Oh, by the way! My phone is next to the shirt just now! Help me find it, thanks." The end of this round of human-computer dialogue will trigger the sliding window to move to select the user's historical voice commands before the user said the above current voice command, and store the corresponding key semantic factors in the sliding window. At this time, the key semantic factors stored in the sliding window include:

[0118] "target: shirt; semantic reference: white shirt on the couch"

[0119] “Target: cup; semantic reference: the red cup on the table”

[0120] "Target: flowers; semantic reference: the dying flowers on the balcony"

[0121] The initial structured data corresponding to the current voice command obtained through semantic analysis of the intent recognition model is: "target: mobile phone; relation triple: mobile phone is next to the shirt; attribute: white shirt". Figure 5 As shown, the item node "mobile phone", the item node "shirt" and the attribute node "white" are generated in the semantic topology graph, and a line is connected between the item node "mobile phone" and the item node "shirt", and a line is connected between the item node "shirt" and the attribute node "white".

[0122] Furthermore, the current semantic factor in the target structured data is matched with the key semantic factor stored in the sliding window, and the key semantic factor "target: shirt; semantic reference: white shirt on the sofa" is matched with the current semantic factor "relation triple: mobile phone is next to the shirt" and "attribute: white shirt" respectively. Figure 5 As shown, in the semantic topology diagram generated according to the current semantic factor, the object node "sofa" corresponding to the above key semantic factor "semantic reference: the white shirt on the sofa" is supplemented, and the object node "sofa" is connected with the object node "shirt" with the arrow pointing to "shirt" to indicate the supporting relationship of "sofa" to "shirt", and the final semantic reference is obtained as "the mobile phone is next to the white shirt on the sofa".

[0123] Therefore, in the process of constructing the semantic topology map, the embodiment of the present application introduces context information to enhance the semantic understanding of the user's current voice command. Therefore, the present application intelligently supplements the current semantic factors through context information, so that it can effectively adapt to the situation where the current voice command has unclear reference, semantic disconnection or strong context dependence, thereby reducing the dependence on the user's complete description, making the semantic understanding more accurate and smooth, and greatly improving the user experience. In addition, the embodiment of the present application uses a sliding window to accurately select and store possibly related context information. The user can configure the sliding window according to their own needs to dynamically adjust the selection range of historical structured data, thereby improving the flexibility and configurability of context information management.

[0124] In addition, the embodiment of the present application realizes the structured expression of the current voice command through the semantic topology map, and can accurately and clearly express the information logic in the current voice command in combination with contextual information, thereby realizing the conversion from spoken expression to structured semantic data, and providing a reliable foundation for the subsequent determination of the target object that the user actually intends to refer to through semantic reasoning, and accurately locating the target object in the image.

[0125] In some embodiments of the present application, the above step S7 includes:

[0126] Extract semantic features of semantic reference and obtain semantic feature vector;

[0127] Perform image feature extraction on the image to be positioned and obtain an image feature vector;

[0128] The semantic feature vector is matched with the image feature vector, and the target object is determined based on the matching result.

[0129] Specifically, in some embodiments of the present application, a language feature extractor is used to extract the semantic feature vector of the semantic reference, and an image feature vector of the image to be located is extracted by an image feature extractor such as Vision-Transformer, and then the visual features and language features are aligned in the feature space. Since the semantic reference obtained by the above steps S1 to S5 effectively filters out the colloquial noise information, the efficiency and accuracy of the feature alignment process are improved. Finally, the high-dimensional features obtained by alignment are output in the output form (bounding box or segmentation) required by the task through a specific neural network output head.

[0130] Accordingly, if Figure 6 As shown, an embodiment of the present application provides a structured visual positioning system based on intent recognition, the system comprising:

[0131] An acquisition module 100 is used to acquire the image to be located and the current voice command of the user;

[0132] The intention recognition module 200 is used to input the current voice command into a pre-trained intention recognition model for semantic structured processing to construct a semantic topology map; wherein the semantic topology map is used to describe the target structured data corresponding to the current voice command;

[0133] The reference reasoning module 300 is used to perform reasoning based on the semantic topology graph to obtain the semantic reference corresponding to the current voice command;

[0134] The feature alignment module 400 is used to perform feature alignment on the semantic reference and the image to be located, so as to locate the target object matching the semantic reference in the image to be located.

[0135] The further functional description of each of the above modules and units is the same as that of the above corresponding embodiments and will not be repeated here.

[0136] The structured visual positioning system in this embodiment is presented in the form of a functional unit, where the unit refers to an ASIC (Application Specific Integrated Circuit) circuit, a processor and memory that executes one or more software or fixed programs, and / or other devices that can provide the above functions.

[0137] The structured visual positioning system proposed in the embodiment of the present application performs deep semantic analysis and structural processing on the voice commands issued by the user through the intention recognition model, thereby constructing a semantic topological map that can accurately describe the user's intention, and obtains semantic references through logical reasoning of the semantic topological map to achieve accurate and rapid understanding of the user's semantic intentions. Finally, the obtained semantic references are aligned with the visual features in the image through multimodal features, so as to accurately locate the target object corresponding to the semantics of the user's command in the image to be located. Therefore, the embodiment of the present application can effectively and accurately parse voice commands containing a lot of colloquial noise, ensure that the user's true intentions can be efficiently and accurately identified in a complex language environment, thereby making the multimodal fusion in the visual positioning process more stable and efficient, greatly improving the semantic understanding ability and robustness, and effectively improving the user experience.

[0138] Thirdly, Figure 7 As shown, an embodiment of the present application provides a computer device, including:

[0139] The memory 20 and the processor 10 are communicatively connected to each other, the memory 20 stores computer instructions, and the processor 10 executes the structured visual positioning method based on intent recognition described in the above embodiment by executing the computer instructions.

[0140] like Figure 7As shown, the computer device includes: one or more processors 10, a memory 20, and interfaces for connecting various components, including high-speed interfaces and low-speed interfaces. Various components are connected to each other using different buses for communication, and can be installed on a common mainboard or installed in other ways as needed. The processor can process the instructions executed in the computer device, including instructions stored in or on the memory to display the graphical information of the GUI on an external input / output device (such as, a display device coupled to the interface). In some optional embodiments, if necessary, multiple processors and / or multiple buses can be used together with multiple memories and multiple memories. Similarly, multiple computer devices can be connected, and each device provides some necessary operations (for example, as a server array, a group of blade servers, or a multi-processor system). Figure 7 A processor 10 is taken as an example.

[0141] The processor 10 may be a central processing unit, a network processor or a combination thereof. The processor 10 may further include a hardware chip. The hardware chip may be a dedicated integrated circuit, a programmable logic device or a combination thereof. The programmable logic device may be a complex programmable logic device, a field programmable gate array, a general purpose array logic or any combination thereof.

[0142] The memory 20 stores instructions executable by at least one processor 10, so that the at least one processor 10 executes the method shown in the above embodiment.

[0143] The memory 20 may include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function; the data storage area may store data created according to the use of the computer device, etc. In addition, the memory 20 may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some optional embodiments, the memory 20 may optionally include a memory remotely arranged relative to the processor 10, and these remote memories may be connected to the computer device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0144] The memory 20 may include a volatile memory, such as a random access memory; the memory may also include a non-volatile memory, such as a flash memory, a hard disk or a solid state drive; the memory 20 may also include a combination of the above types of memory.

[0145] The computer device further comprises a communication interface 30 for the computer device to communicate with other devices or a communication network.

[0146] The embodiment of the present application also provides a computer-readable storage medium. The above method according to the embodiment of the present application can be implemented in hardware, firmware, or can be implemented as a computer code that can be recorded in a storage medium, or can be implemented as a computer code that is originally stored in a remote storage medium or a non-temporary machine-readable storage medium and will be stored in a local storage medium through a network download, so that the method described herein can be stored in such software processing on a storage medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware. Among them, the storage medium can be a magnetic disk, an optical disk, a read-only storage memory, a random access memory, a flash memory, a hard disk or a solid-state hard disk, etc.; further, the storage medium can also include a combination of the above types of memories. It can be understood that a computer, a processor, a microprocessor controller, or programmable hardware includes a storage component that can store or receive software or computer code. When the software or computer code is accessed and executed by a computer, a processor, or hardware, the method shown in the above embodiment is implemented.

[0147] The embodiment of the present application provides a computer program product, which includes computer instructions, which are stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes the method of any embodiment of the present application.

[0148] Although the embodiments of the present application are described in conjunction with the accompanying drawings, those skilled in the art may make various modifications and variations without departing from the spirit and scope of the present application, and such modifications and variations are all within the scope defined by the appended claims.

[0149] The systems, devices, modules or units described in the above embodiments may be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer. Specifically, the computer may be, for example, a personal computer, a laptop computer, a cellular phone, a camera phone, a smart phone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, a wearable device, or a combination of any of these devices.

[0150] For the convenience of description, the above device is described in various units according to their functions. Of course, when implementing the present application, the functions of each unit can be implemented in the same or multiple software and / or hardware.

[0151] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.

[0152] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0153] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0154] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0155] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0156] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0157] The above is only an embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included in the scope of the claims of the present application.

[0158] Although the embodiments of the present application have been described in conjunction with the accompanying drawings, those skilled in the art may make various modifications and variations without departing from the spirit and scope of the present application, and such modifications and variations are all within the scope defined by the appended claims.

Claims

1. A structured visual positioning method based on intention recognition, characterized in that: The method comprises: Obtain the image to be located and the user's current voice command; Inputting the current voice command into a pre-trained intention recognition model for semantic structuring processing to construct a semantic topology map; wherein the semantic topology map is used to describe the target structured data corresponding to the current voice command; Performing reasoning based on the semantic topology graph to obtain the semantic reference corresponding to the current voice instruction; Feature alignment is performed on the semantic reference and the image to be located, so as to locate a target object matching the semantic reference in the image to be located.

2. The method according to claim 1, characterized in that The inputting the current voice command into a pre-trained intention recognition model for semantic structural processing to construct a semantic topology map includes: Performing semantic analysis on the current voice instruction by using the intention recognition model to obtain a current semantic factor in the current voice instruction; Outputting according to a preset data format based on the current semantic factor to obtain the semantic topology map; The semantic factors include the task, target, relation triples and attributes corresponding to the current voice instruction.

3. The method according to claim 1 or 2, characterized in that: The training process of the intent recognition model includes: Determining a preset number of first corpora in a specialized corpus of visual positioning descriptions; wherein the specialized corpus is constructed based on historical voice instructions; In response to the labeling operation on the preset number of first corpora, a labeling sample is obtained; Based on the annotated samples, automatically annotate the second corpora other than the first corpus in the specialized corpus, and filter and screen the automatically annotated second corpora to construct a candidate sample set using the filtered second corpora and the first corpus; The candidate sample set is proportionally adjusted according to the labeled categories to obtain a training sample set, and the initial recognition model is trained using the training sample set to obtain the intent recognition model.

4. The method according to claim 3, characterized in that The automatic annotation includes: Determine the prompt words according to the requirements of the annotation task and output format; The second corpus is annotated using the CoT technology according to the prompt word and the annotated sample.

5. The method according to claim 2, characterized in that: Before outputting the current semantic factor in a preset data format to obtain the semantic topology map, the method further includes: Using a sliding window method, determine the historical structured data corresponding to the occurrence time of the current voice command, select key semantic factors from the semantic factors of the historical structured data, and store the key semantic factors as context information; The outputting according to a preset data format based on the current semantic factor to obtain the semantic topology map includes: The semantic topology map is output according to the current semantic factor and the context information in a preset data format.

6. The method according to claim 5, characterized in that The step of outputting the semantic topology map in a preset data format according to the current semantic factor and the context information includes: Determining initial structured data corresponding to the current semantic factor; The context information is matched with the initial structured data, and the initial structured data is contextually supplemented according to the matching result, so as to obtain the semantic topology map in the preset data format.

7. The method according to claim 2, characterized in that The reasoning based on the semantic topology graph to obtain the semantic reference corresponding to the current voice instruction includes: Perform reasoning based on the target in the current voice instruction and the semantic topology graph to obtain attribute and relationship triples related to the target; The semantic reference is generated according to the attribute and relationship triples related to the target.

8. The method according to claim 1, characterized in that The step of aligning features of the semantic reference and the image to be located to locate a target object in the image to be located that matches the semantic reference includes: Extracting semantic features from the semantic reference and obtaining a semantic feature vector; Extracting image features from the image to be located and obtaining an image feature vector; The semantic feature vector is matched with the image feature vector, and the target object is determined according to the matching result.

9. A structured visual positioning system based on intention recognition, characterized in that: The system comprises: An acquisition module is used to acquire the image to be located and the user's current voice command; An intention recognition module, used for inputting the current voice instruction into a pre-trained intention recognition model for semantic structural processing to construct a semantic topological map; wherein the semantic topological map is used to describe the target structured data corresponding to the current voice instruction; A reference reasoning module, used for performing reasoning based on the semantic topology graph to obtain the semantic reference corresponding to the current voice instruction; The feature alignment module is used to perform feature alignment on the semantic reference and the image to be located, so as to locate a target object matching the semantic reference in the image to be located.

10. A computer device, characterized in that: include: A memory and a processor, wherein the memory and the processor are communicatively connected to each other, the memory stores computer instructions, and the processor executes the structured visual positioning method based on intent recognition as described in any one of claims 1 to 8 by executing the computer instructions.

Images