Intention recognition method, device, equipment and storage medium based on multimodal model

Abstract

The present application relates to the technical field of intention recognition, and provides an intention recognition method, device, equipment and storage medium based on a multimodal model. The method includes: obtaining text instruction information and a target video; inputting the text instruction information into a preset text feature extraction model to obtain text instruction feature information; performing frame processing on the target video to obtain a video frame sequence corresponding to the target video; inputting the video frame sequence into a preset visual feature extraction model to obtain visual feature information; fusing the text instruction feature information with the visual feature information to obtain fused feature information, and inputting the fused feature information into a preset multimodal model to obtain the user intention text. When this method is applied to a robot, it can reduce the robot's dependence on frequent human instructions in complex tasks, thereby achieving smoother and more efficient human-machine collaboration.

Description

Technical Field

[0001] The present application relates to the technical field of intent recognition, and in particular to a method, apparatus, device and storage medium for intent recognition based on a multimodal model. Background Art

[0002] When the idea of ​​incorporating natural language commands into robot control strategies is applied to human-robot collaboration, human users may need to engage in dialogue with the robot at every step of a long-term task. This rarely happens in human-to-human collaboration, as humans are able to rely on their shared knowledge of the task to track their partners' progress. This difference stems from the accumulation of tacit understanding and common experience between humans, which allows them to collaborate efficiently without frequent communication. However, robots currently lack this common knowledge and tacit understanding, and must rely on frequent natural language communication to ensure the correct execution of tasks and progress tracking. This reflects the limitations of current robots in understanding and predicting human intentions, and it is necessary to further improve the autonomy and intelligence of robots in complex tasks to reduce dependence on frequent human instructions, thereby achieving more smooth and efficient human-robot collaboration. Summary of the invention

[0003] The present application provides a method, device, equipment and storage medium for intention recognition based on a multimodal model to solve the problems raised by the above background technology.

[0004] In a first aspect, the present application provides an intention recognition method based on a multimodal model, comprising:

[0005] Get text instruction information and target video;

[0006] Inputting the text instruction information into a preset text feature extraction model to obtain text instruction feature information;

[0007] Performing frame processing on the target video to obtain a video frame sequence corresponding to the target video;

[0008] Inputting the video frame sequence into a preset visual feature extraction model to obtain visual feature information;

[0009] The text instruction feature information is fused with the visual feature information to obtain fused feature information, and the fused feature information is input into a preset multimodal model to obtain a user intention text.

[0010] In a possible implementation, the obtaining of text instruction information and target video includes:

[0011] In response to a user inputting the text instruction information, acquiring the text instruction information;

[0012] After the text instruction information is acquired, the user's operation image is acquired through a preset camera device until the user stops the operation.

[0013] In a possible implementation, inputting the video frame sequence into a preset visual feature extraction model to obtain visual feature information includes:

[0014] Inputting the video frame sequence into a preset visual feature extraction model; the visual feature extraction model includes an image feature extraction layer, a 3D convolution layer, a multi-mechanism temporal enhancement layer and a learnable projection layer;

[0015] The image feature extraction layer sequentially extracts features from each video frame in the video frame sequence to obtain an image feature sequence, and inputs the image feature sequence into the 3D convolution layer;

[0016] The 3D convolution layer sequentially performs convolution processing on each image feature in the image feature sequence to obtain a convolution feature sequence, and inputs the convolution feature sequence into the multi-mechanism temporal enhancement layer;

[0017] The multi-mechanism temporal enhancement layer performs temporal enhancement processing on the convolution feature sequence to obtain initial visual feature information, and inputs the initial visual feature information into the learnable projection layer;

[0018] The learnable projection layer projects the initial visual feature information to obtain the visual feature information.

[0019] In a possible implementation, the multimodal model training method includes:

[0020] Acquire a multi-task training data set; the multi-task training data set includes training data of multiple tasks;

[0021] Constructing an expert mixture model; the expert mixture model includes a task indicator, a plurality of expert sub-models and a basic intent recognition model;

[0022] The model parameters of the basic intent recognition model are frozen in the expert mixture model, and multi-task training is performed on the expert mixture model based on the training data of the multiple tasks to obtain the multimodal model.

[0023] In a possible implementation, performing multi-task training on the expert mixture model based on the training data of the multiple tasks to obtain the multimodal model includes:

[0024] For each of the tasks, a low-rank adaptation method is used to train the expert sub-model corresponding to the task using the training data corresponding to the task, so as to obtain a low-rank adaptation weight corresponding to the expert sub-model;

[0025] training the task indicator based on the training data of the plurality of tasks;

[0026] For each of the expert sub-models, initializing the expert sub-model based on the low-rank adaptive weight corresponding to the expert sub-model to obtain an intermediate multimodal model;

[0027] The intermediate multimodal model is trained based on the multi-task training data set to obtain the multimodal model.

[0028] In a possible implementation, the step of obtaining a multi-task training data set includes:

[0029] Acquire a plurality of initial training videos; wherein the initial training videos include human behaviors;

[0030] For each of the initial training videos, performing frame processing on the initial training video to obtain a training video frame sequence corresponding to the initial training video;

[0031] For each of the initial training videos, the training video frame sequence corresponding to the initial training video is divided into a first training video frame subsequence and a second training video frame subsequence, and a video consisting of the first training video frame subsequence is determined as a target training video; wherein the second training video frame subsequence is located after the first training video frame subsequence;

[0032] For each of the initial training videos, generating the intent and instruction corresponding to the initial training video based on the training video frame sequence corresponding to the initial training video;

[0033] For each of the initial training videos, determining task information corresponding to the initial training video based on scene information corresponding to the initial training video; wherein the task information includes at least one task;

[0034] For each of the initial training videos, a mapping relationship is constructed between a target training video corresponding to the initial training video and the intention, instruction, and task information corresponding to the initial training video to obtain the multi-task training data set.

[0035] In a possible implementation, the generating the intent and instruction corresponding to the initial training video based on the training video frame sequence corresponding to the initial training video includes:

[0036] Inputting a first initial training video frame of the training video frame sequence into the visual language model to obtain a first video subtitle of the first initial training video frame; wherein the visual language model is a pre-trained neural network model;

[0037] Inputting the first video subtitle and a second initial training video frame of the training video frame sequence into the visual language model to obtain a second video subtitle of the second initial training video frame;

[0038] Inputting the second video subtitle and the third initial training video frame of the training video frame sequence into the visual language model to obtain a third video subtitle of the third initial training video frame;

[0039] Iterate the steps of inputting the first video subtitle and the second initial training video frame of the training video frame sequence into the visual language model to obtain the second video subtitle of the second initial training video frame until the video subtitle of the last initial training video frame of the training video frame sequence is obtained;

[0040] Inputting each video subtitle into a preset large language model in sequence to obtain first language description information, and generating the instruction based on the first language description information; wherein the large language model is a pre-trained neural network model;

[0041] The video subtitles corresponding to each initial training video frame in the second training video frame subsequence are sequentially input into the large language model to obtain second language description information, and the intent is generated based on the second language description information.

[0042] In a second aspect, the present application provides an intention recognition device based on a multimodal model, comprising:

[0043] An acquisition module is used to acquire text instruction information and target video;

[0044] A first input module, used for inputting the text instruction information into a preset text feature extraction model to obtain text instruction feature information;

[0045] A frame processing module, used to perform frame processing on the target video to obtain a video frame sequence corresponding to the target video;

[0046] A second input module is used to input the video frame sequence into a preset visual feature extraction model to obtain visual feature information;

[0047] The fusion module is used to fuse the text instruction feature information with the visual feature information to obtain fused feature information, and input the fused feature information into a preset multimodal model to obtain the user intention text.

[0048] In a third aspect, the present application provides a terminal device, comprising a processor, a memory, and a computer program stored in the memory and executable by the processor, wherein when the computer program is executed by the processor, the multimodal model-based intent recognition method as described in any one of the above items is implemented.

[0049] In a fourth aspect, the present application provides a computer-readable storage medium, on which a computer program is stored, wherein when the computer program is executed by a processor, the multimodal model-based intent recognition method as described in any one of the above items is implemented.

[0050] The present application provides an intention recognition method, device, equipment and storage medium based on a multimodal model. The method includes: obtaining text instruction information and a target video; inputting the text instruction information into a preset text feature extraction model to obtain text instruction feature information; performing frame processing on the target video to obtain a video frame sequence corresponding to the target video; inputting the video frame sequence into a preset visual feature extraction model to obtain visual feature information; fusing the text instruction feature information with the visual feature information to obtain fused feature information, and inputting the fused feature information into a preset multimodal model to obtain a user intention text. On the one hand, the method ensures the diversity and richness of the input data by obtaining text instruction information and a target video, and extracting features from the text instruction information and the target video respectively, which helps to capture task-related detail information. On the other hand, the text feature extraction model and the visual feature extraction model process the text instruction and the video frame sequence respectively to generate high-quality feature information, so that the multimodal model can more comprehensively understand the semantics and content of the input data. This multi-level, multimodal feature extraction method improves the information capture capability of the model. On the other hand, by fusing text instruction feature information with visual feature information to generate fused feature information, effective integration of multimodal data is achieved, and the model's understanding and reasoning capabilities in complex scenarios are enhanced. When the method of this embodiment is applied to a robot, the robot can reduce its reliance on frequent human instructions in complex tasks, thereby achieving smoother and more efficient human-machine collaboration. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying any creative work.

[0052] Figure 1 A flowchart of a method for identifying intentions based on a multimodal model provided in an embodiment of the present application;

[0053] Figure 2 A schematic block diagram of the structure of an intention recognition device based on a multimodal model provided in an embodiment of the present application;

[0054] Figure 3 A schematic block diagram of the structure of a terminal device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0055] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0056] The flowcharts shown in the accompanying drawings are only examples and do not necessarily include all the contents and operations / steps, nor must they be executed in the order described. For example, some operations / steps may also be decomposed, combined or partially merged, so the actual execution order may change according to actual conditions.

[0057] It should also be understood that the terms used in this application specification are only for the purpose of describing specific embodiments and are not intended to limit the application. As used in this application specification and the appended claims, unless the context clearly indicates otherwise, the singular forms "a", "an" and "the" are intended to include plural forms.

[0058] It should be further understood that the term “and / or” used in the specification and appended claims refers to any combination and all possible combinations of one or more of the associated listed items, and includes these combinations.

[0059] When the concept of incorporating natural language commands into robot control strategies is applied to human-robot collaboration, human users may need to communicate with the robot at every step of a long-term task. This situation rarely occurs in human-to-human collaboration, because humans are able to rely on their shared knowledge of the task to track the progress of their partners. This difference stems from the accumulation of tacit understanding and common experience between humans, which allows them to collaborate efficiently without frequent communication. However, robots currently lack this common knowledge and tacit understanding, and must ensure the correct execution of tasks and progress tracking through frequent natural language communication. This reflects the limitations of current robots in understanding and predicting human intentions, and it is necessary to further improve the autonomy and intelligence level of robots in complex tasks to reduce dependence on frequent human instructions, thereby achieving smoother and more efficient human-robot collaboration. To this end, the present application provides an intention recognition method, device, equipment and storage medium based on a multimodal model to solve the above problems.

[0060] In conjunction with the accompanying drawings, some embodiments of the present application are described in detail below. In the absence of conflict, the following embodiments and features in the embodiments can be combined with each other.

[0061] See also Figure 1 , Figure 1 A flowchart of the multimodal model-based intent recognition method provided in the embodiment of the present application is shown in FIG. Figure 1 As shown, the method includes steps S1 to S5.

[0062] It should be noted that the method of this embodiment is applied to a robot. The executor of this embodiment may be an intention recognition device based on a multimodal model, or may be a terminal or a server. The following description will be made using a server as an example.

[0063] Step S1, obtaining text instruction information and target video.

[0064] Specifically, step S1 includes: in response to the user's input of the text instruction information, acquiring the text instruction information; after acquiring the text instruction information, acquiring the user's operation image through a preset camera device until the user stops the operation.

[0065] It should be noted that after the user inputs the text instruction information, the user performs relevant preparations based on the tasks in the text instruction information. During the preparation process, the server controls the preset camera to shoot the user's operation process until the user stops the operation. For example, after the user inputs the text instruction information for making lettuce with oyster sauce, the user takes the lettuce out of the refrigerator and places it on the operating table. After receiving the text instruction information for making lettuce with oyster sauce, the server controls the camera to shoot the process of the user taking the lettuce out of the refrigerator and placing it on the operating table, so as to obtain the user's operation image.

[0066] Step S2: input the text instruction information into a preset text feature extraction model to obtain text instruction feature information.

[0067] The text feature extraction model is a neural network model that has been trained in advance, and the neural network model includes a feature extraction layer, a convolution layer and a learnable projection layer.

[0068] Specifically, step S2 includes: the feature extraction layer performs feature extraction on the text instruction information to obtain initial text instruction feature information; the convolution layer performs convolution processing on the initial text instruction feature information to obtain intermediate text instruction feature information; the learnable projection layer projects the intermediate text instruction feature information to obtain the text instruction feature information.

[0069] It can be understood that the text feature extraction model in step S2 can effectively extract rich and semantically meaningful text instruction feature information from the text instruction information through a learnable feature extraction method, so that the generated text instruction feature information and the input information of the multimodal model are in the same feature space, providing high-quality input for subsequent intent recognition, thereby improving the accuracy of intent recognition.

[0070] Step S3: performing frame processing on the target video to obtain a video frame sequence corresponding to the target video.

[0071] It should be noted that the time interval between two adjacent video frames in the video frame sequence is no greater than 0.5S.

[0072] Step S4: input the video frame sequence into a preset visual feature extraction model to obtain visual feature information.

[0073] Specifically, step S4 includes: inputting the video frame sequence into a preset visual feature extraction model; the visual feature extraction model includes an image feature extraction layer, a 3D convolution layer, a multi-mechanism temporal enhancement layer and a learnable projection layer; the image feature extraction layer sequentially extracts features from each video frame in the video frame sequence to obtain an image feature sequence, and inputs the image feature sequence into the 3D convolution layer; the 3D convolution layer sequentially convolves each image feature in the image feature sequence to obtain a convolution feature sequence, and inputs the convolution feature sequence into the multi-mechanism temporal enhancement layer; the multi-mechanism temporal enhancement layer performs temporal enhancement processing on the convolution feature sequence to obtain initial visual feature information, and inputs the initial visual feature information into the learnable projection layer; the learnable projection layer projects the initial visual feature information to obtain the visual feature information.

[0074] It can be understood that, in step S4, on the one hand, by using a multi-level visual feature extraction model, the spatial and temporal information in the video frame sequence is fully captured. This hierarchical design enables the model to gradually extract visual features from low-level to high-level, so as to fully understand the video content; on the other hand, by introducing the 3D convolution layer, the synchronous processing of the spatial and temporal dimension information in the video frame sequence is realized, and the effect of effectively integrating spatiotemporal information is achieved. This processing method can capture the dynamic change characteristics in the video and provide richer information for subsequent intent recognition tasks; on the other hand, by adopting a multi-mechanism timing enhancement layer, the effect of enhancing timing dependence is achieved. This design enables the model to better understand and represent complex timing patterns, especially when processing long video sequences; in addition, by introducing a learnable projection layer, the effect of adaptively optimizing the visual feature information is achieved, which not only improves the expressiveness of the features, but also makes the generated visual features and the input information of the multimodal model in the same feature space, laying the foundation for subsequent intent recognition.

[0075] Step S5: Fusing the text instruction feature information with the visual feature information to obtain fused feature information, and inputting the fused feature information into a preset multimodal model to obtain a user intention text.

[0076] Specifically, a multi-head attention mechanism is used to fuse the text instruction information with the visual feature information to obtain fused feature information.

[0077] Among them, the training method of the multimodal model includes: obtaining a multi-task training data set; the multi-task training data set includes training data of multiple tasks; constructing an expert mixture model; the expert mixture model includes a task indicator, multiple expert sub-models and a basic intent recognition model; freezing the model parameters of the basic intent recognition model in the expert mixture model, and performing multi-task training on the expert mixture model based on the training data of the multiple tasks to obtain the multimodal model.

[0078] It can be understood that the above-mentioned multimodal model training method, by freezing the model parameters of the basic intent recognition model, can ensure that when the expert mixture model is trained, the expert mixture model will not forget the existing historical knowledge.

[0079] Wherein, the obtaining of the multi-task training data set includes: obtaining a number of initial training videos, wherein the initial training video includes human behavior, specifically, obtaining a video including human behavior through Internet data, and using the obtained video as the initial training video; for each of the initial training videos, performing frame processing on the initial training video to obtain a training video frame sequence corresponding to the initial training video; for each of the initial training videos, dividing the training video frame sequence corresponding to the initial training video into a first training video frame subsequence and a second training video frame subsequence, and determining that the video composed of the first training video frame subsequence is a target training video; wherein the second training video frame subsequence is located after the first training video frame subsequence; for each of the initial training videos, generating the intent and instruction corresponding to the initial training video based on the training video frame sequence corresponding to the initial training video; for each of the initial training videos, determining the task information corresponding to the initial training video based on the scene information corresponding to the initial training video; wherein the task information includes at least one task; for each of the initial training videos, constructing a mapping relationship between the target training video corresponding to the initial training video and the intent, instruction and task information corresponding to the initial training video, to obtain the multi-task training data set.

[0080] It should be noted that the task information includes one or more of visual description, region-level reference expression understanding, reference expression generation, multi-object detection, relationship reasoning, and human intention prediction. The task information corresponding to the initial training video is determined according to the scene information of the initial training video. For example, the scene of the initial training video is preparing coffee indoors, and the task information corresponding to the initial training video includes visual description, i.e., performing visual feature extraction and region-level reference expression understanding on the target training video corresponding to the initial training video, i.e., performing position information extraction and human intention prediction on the characters in each video frame of the target training video corresponding to the initial training video, i.e., the intention corresponding to the initial training video; for another example, the scene of the initial training video is driving a vehicle on the road, and the task information corresponding to the initial training video includes visual description, i.e., performing visual feature extraction and relational reasoning on the target training video corresponding to the initial training video, i.e., performing feature extraction and reference expression generation on the positional relationship between each thing (including objects and people) in each video frame of the target training video corresponding to the initial training video, i.e., marking the dangerous behavior of pedestrians in each video frame of the target training video corresponding to the initial training video with a position box, and performing feature extraction and multi-object detection on the marked content, i.e., performing feature extraction and human intention prediction on each thing (including objects and people) in each video frame of the target training video corresponding to the initial training video, i.e., the intention corresponding to the initial training video.

[0081] Wherein, the generating of the intent and instruction corresponding to the initial training video based on the training video frame sequence corresponding to the initial training video includes: inputting the first initial training video frame of the training video frame sequence into the visual language model to obtain the first video subtitle of the first initial training video frame, wherein the visual language model is a pre-trained neural network model, the video language model includes the CoT process of the Wolf algorithm, and the first video subtitle includes the scene-level information of the first initial training video frame and the position of each object; inputting the first video subtitle and the second initial training video frame of the training video frame sequence into the visual language model to obtain the second video subtitle of the second initial training video frame, wherein the second video subtitle includes the scene-level information of the second initial training video frame and the position of each object; inputting the second video subtitle and the third initial training video frame of the training video frame sequence into the visual language model The method comprises the steps of inputting the first video subtitle and the second initial training video frame of the training video frame sequence into the visual language model to obtain the second video subtitle of the second initial training video frame, until the video subtitle of the last initial training video frame of the training video frame sequence is obtained; inputting each video subtitle into a preset large language model in sequence to obtain first language description information, and generating the instruction based on the first language description information, wherein the large language model is a pre-trained neural network model; inputting the video subtitles corresponding to each initial training video frame in the second training video frame subsequence into the large language model in sequence to obtain second language description information, and generating the intent based on the second language description information.

[0082] It can be understood that the above method of obtaining a multi-task training data set, on the one hand, can obtain a rich and diverse data source by obtaining initial training videos containing human behaviors from the Internet, thereby enhancing the diversity of the training data set and achieving the effect of covering a variety of scenarios and behavior patterns, which is helpful to accurately predict human intentions in complex and changing environments, improve the reliability of intention prediction, and enhance the generalization and reliability of the model. On the other hand, through video frame processing and subsequence segmentation, long videos are decomposed into smaller time segments, which helps to capture dynamic changes in the video in a fine-grained manner and improve the accuracy of the multi-task training data set. On the other hand, by generating video subtitles based on a visual language model, scene-level information and the location of each object in the video can be automatically extracted, providing structured descriptive information, thereby reducing the workload of manual labeling and improving data processing efficiency.

[0083] Among them, the multi-task training of the expert mixture model based on the training data of the multiple tasks to obtain the multimodal model includes: for each of the tasks, using the low-rank adaptation method (English abbreviation: LORA) to train the expert sub-model corresponding to the task through the training data corresponding to the task, to obtain the low-rank adaptation weight corresponding to the expert sub-model; training the task indicator based on the training data of the multiple tasks; for each of the expert sub-models, initializing the expert sub-model based on the low-rank adaptation weight corresponding to the expert sub-model to obtain an intermediate multimodal model; training the intermediate multimodal model based on the multi-task training data set to obtain the multimodal model.

[0084] It can be understood that the above-mentioned method of performing multi-task training on the expert mixture model based on the training data of the multiple tasks, on the one hand, effectively fine-tunes the expert mixture model by adopting a low-rank adaptation method, thereby reducing the number of parameters to be adjusted, thereby reducing the risk of overfitting. The introduction of this low-rank adaptation method enables the multimodal model to adapt quickly when processing new tasks without the need to adjust a large number of additional parameters, thereby improving the training efficiency and adaptability of the multimodal model.

[0085] The method provided in this embodiment, on the one hand, by obtaining text instruction information and target video, and extracting features from the text instruction information and target video respectively, ensures the diversity and richness of input data, and helps to capture task-related detail information. On the other hand, the text feature extraction model and the visual feature extraction model process the text instruction and the video frame sequence respectively, generate high-quality feature information, so that the multimodal model can understand the semantics and content of the input data more comprehensively. This multi-level, multimodal feature extraction method improves the information capture ability of the model. On the other hand, by fusing the text instruction feature information with the visual feature information to generate fused feature information, the effective integration of multimodal data is achieved, and the understanding and reasoning ability of the model in complex scenes is enhanced. When the method of this embodiment is applied to a robot, the robot can reduce its dependence on frequent human instructions in complex tasks, thereby achieving smoother and more efficient human-machine collaboration.

[0086] See also Figure 2 , Figure 2 A schematic block diagram of the structure of the multimodal model-based intention recognition device 100 provided in an embodiment of the present application is shown in FIG. Figure 2 As shown, the intention recognition device based on the multimodal model provided in the embodiment of the present application includes:

[0087] The acquisition module 110 is used to acquire text instruction information and target video.

[0088] The first input module 120 is used to input the text instruction information into a preset text feature extraction model to obtain text instruction feature information.

[0089] The frame processing module 130 is used to perform frame processing on the target video to obtain a video frame sequence corresponding to the target video.

[0090] The second input module 140 is used to input the video frame sequence into a preset visual feature extraction model to obtain visual feature information.

[0091] The fusion module 150 is used to fuse the text instruction feature information with the visual feature information to obtain fused feature information, and input the fused feature information into a preset multimodal model to obtain the user intention text.

[0092] It should be noted that technical personnel in the relevant technical field can clearly understand that, for the convenience and conciseness of description, the specific working process of the above-described device and each module can refer to the corresponding process in the aforementioned embodiment of the intention recognition method based on the multimodal model, and will not be repeated here.

[0093] The multimodal model-based intention recognition device 100 provided in the above embodiment can be implemented in the form of a computer program. The computer program can be used in Figure 3 The system is run on the terminal device 200 shown.

[0094] See also Figure 3 , Figure 3 The terminal device 200 provided in the embodiment of the present application is a schematic block diagram of the structure, wherein the terminal device 200 includes a processor 201 and a memory 202, wherein the processor 201 and the memory 202 are connected via a system bus 203, wherein the memory 202 may include a non-volatile storage medium and an internal memory.

[0095] The non-volatile storage medium may store a computer program. The computer program includes program instructions, and when the program instructions are executed by the processor 201, the processor 201 may execute any of the above-mentioned intention recognition methods based on the multimodal model.

[0096] The processor 201 is used to provide computing and control capabilities to support the operation of the entire terminal device 200.

[0097] The internal memory provides an environment for the operation of the computer program in the non-volatile storage medium. When the computer program is executed by the processor 201, the processor 201 can execute any of the above-mentioned intention recognition methods based on the multimodal model.

[0098] Those skilled in the art will understand that Figure 3The structure shown in the figure is only a block diagram of a partial structure related to the scheme of the present application, and does not constitute a limitation on the terminal device 200 involved in the scheme of the present application. The specific terminal device 200 may include more or fewer components than shown in the figure, or combine certain components, or have a different component arrangement.

[0099] It should be understood that the processor 201 may be a central processing unit (CPU), and the processor 201 may also be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. Among them, the general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0100] In some embodiments, the processor 201 is used to run a computer program stored in the memory to implement the following steps:

[0101] Get text instruction information and target video;

[0102] Inputting the text instruction information into a preset text feature extraction model to obtain text instruction feature information;

[0103] Performing frame processing on the target video to obtain a video frame sequence corresponding to the target video;

[0104] Inputting the video frame sequence into a preset visual feature extraction model to obtain visual feature information;

[0105] The text instruction feature information is fused with the visual feature information to obtain fused feature information, and the fused feature information is input into a preset multimodal model to obtain a user intention text.

[0106] It should be noted that technical personnel in the relevant field can clearly understand that, for the convenience and conciseness of description, the specific working process of the terminal device 200 described above can refer to the corresponding process of the aforementioned intention recognition method based on the multimodal model, and will not be repeated here.

[0107] An embodiment of the present application also provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by one or more processors, the one or more processors implement the multimodal model-based intent recognition method provided in the embodiment of the present application.

[0108] The computer-readable storage medium may be an internal storage unit of the terminal device 200 in the aforementioned embodiment, such as a hard disk or memory of the terminal device 200. The computer-readable storage medium may also be an external storage device of the terminal device 200, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. equipped with the terminal device 200.

[0109] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed in the present application, and these modifications or replacements should be included in the protection scope of the present application. Therefore, the protection scope of the present application shall be based on the protection scope of the claims.

Claims

1. An intention recognition method based on a multimodal model, characterized in that: include: Get text instruction information and target video; Inputting the text instruction information into a preset text feature extraction model to obtain text instruction feature information; Performing frame processing on the target video to obtain a video frame sequence corresponding to the target video; Inputting the video frame sequence into a preset visual feature extraction model to obtain visual feature information; Fusing the text instruction feature information with the visual feature information to obtain fused feature information, and inputting the fused feature information into a preset multimodal model to obtain a user intention text; Wherein, the training method of the multimodal model includes: Acquire a multi-task training data set; the multi-task training data set includes training data of multiple tasks; Constructing an expert mixture model; the expert mixture model includes a task indicator, a plurality of expert sub-models and a basic intent recognition model; Freezing the model parameters of the basic intent recognition model in the expert mixture model, and performing multi-task training on the expert mixture model based on the training data of the multiple tasks to obtain the multimodal model; The performing multi-task training on the expert mixture model based on the training data of the multiple tasks to obtain the multimodal model includes: For each of the tasks, a low-rank adaptation method is used to train the expert sub-model corresponding to the task using the training data corresponding to the task, so as to obtain a low-rank adaptation weight corresponding to the expert sub-model; training the task indicator based on the training data of the plurality of tasks; For each of the expert sub-models, initializing the expert sub-model based on the low-rank adaptive weight corresponding to the expert sub-model to obtain an intermediate multimodal model; Training the intermediate multimodal model based on the multi-task training data set to obtain the multimodal model; The obtaining of a multi-task training data set includes: Acquire a plurality of initial training videos; wherein the initial training videos include human behaviors; For each of the initial training videos, performing frame processing on the initial training video to obtain a training video frame sequence corresponding to the initial training video; For each of the initial training videos, the training video frame sequence corresponding to the initial training video is divided into a first training video frame subsequence and a second training video frame subsequence, and a video consisting of the first training video frame subsequence is determined as a target training video; wherein the second training video frame subsequence is located after the first training video frame subsequence; For each of the initial training videos, generating the intent and instruction corresponding to the initial training video based on the training video frame sequence corresponding to the initial training video; For each of the initial training videos, determining task information corresponding to the initial training video based on scene information corresponding to the initial training video; wherein the task information includes at least one task; For each of the initial training videos, construct a mapping relationship between a target training video corresponding to the initial training video and the intention, instruction, and task information corresponding to the initial training video to obtain the multi-task training data set; The generating the intent and instruction corresponding to the initial training video based on the training video frame sequence corresponding to the initial training video includes: Inputting a first initial training video frame of the training video frame sequence into a visual language model to obtain a first video subtitle of the first initial training video frame; wherein the visual language model is a pre-trained neural network model; Inputting the first video subtitle and a second initial training video frame of the training video frame sequence into the visual language model to obtain a second video subtitle of the second initial training video frame; Inputting the second video subtitle and the third initial training video frame of the training video frame sequence into the visual language model to obtain a third video subtitle of the third initial training video frame; Iterate the steps of inputting the first video subtitle and the second initial training video frame of the training video frame sequence into the visual language model to obtain the second video subtitle of the second initial training video frame until the video subtitle of the last initial training video frame of the training video frame sequence is obtained; Inputting each video subtitle into a preset large language model in sequence to obtain first language description information, and generating the instruction based on the first language description information; wherein the large language model is a pre-trained neural network model; The video subtitles corresponding to each initial training video frame in the second training video frame subsequence are sequentially input into the large language model to obtain second language description information, and the intent is generated based on the second language description information.

2. The method for intention recognition based on a multimodal model according to claim 1, characterized in that: The obtaining of text instruction information and target video includes: In response to a user inputting the text instruction information, acquiring the text instruction information; After the text instruction information is acquired, the user's operation image is acquired through a preset camera device until the user stops the operation.

3. The method for intention recognition based on a multimodal model according to claim 1, characterized in that: The step of inputting the video frame sequence into a preset visual feature extraction model to obtain visual feature information includes: Inputting the video frame sequence into a preset visual feature extraction model; the visual feature extraction model includes an image feature extraction layer, a 3D convolution layer, a multi-mechanism temporal enhancement layer and a learnable projection layer; The image feature extraction layer sequentially extracts features from each video frame in the video frame sequence to obtain an image feature sequence, and inputs the image feature sequence into the 3D convolution layer; The 3D convolution layer sequentially performs convolution processing on each image feature in the image feature sequence to obtain a convolution feature sequence, and inputs the convolution feature sequence into the multi-mechanism temporal enhancement layer; The multi-mechanism temporal enhancement layer performs temporal enhancement processing on the convolution feature sequence to obtain initial visual feature information, and inputs the initial visual feature information into the learnable projection layer; The learnable projection layer projects the initial visual feature information to obtain the visual feature information.

4. An intention recognition device based on a multimodal model, characterized in that: include: An acquisition module is used to acquire text instruction information and target video; A first input module, used for inputting the text instruction information into a preset text feature extraction model to obtain text instruction feature information; A frame processing module, used to perform frame processing on the target video to obtain a video frame sequence corresponding to the target video; A second input module is used to input the video frame sequence into a preset visual feature extraction model to obtain visual feature information; A fusion module, used for fusing the text instruction feature information with the visual feature information to obtain fused feature information, and inputting the fused feature information into a preset multimodal model to obtain a user intention text; Wherein, the training method of the multimodal model includes: Acquire a multi-task training data set; the multi-task training data set includes training data of multiple tasks; Constructing an expert mixture model; the expert mixture model includes a task indicator, a plurality of expert sub-models and a basic intent recognition model; Freezing the model parameters of the basic intent recognition model in the expert mixture model, and performing multi-task training on the expert mixture model based on the training data of the multiple tasks to obtain the multimodal model; The performing multi-task training on the expert mixture model based on the training data of the multiple tasks to obtain the multimodal model includes: For each of the tasks, a low-rank adaptation method is used to train the expert sub-model corresponding to the task using the training data corresponding to the task, so as to obtain a low-rank adaptation weight corresponding to the expert sub-model; training the task indicator based on the training data of the plurality of tasks; For each of the expert sub-models, initializing the expert sub-model based on the low-rank adaptive weight corresponding to the expert sub-model to obtain an intermediate multimodal model; Training the intermediate multimodal model based on the multi-task training data set to obtain the multimodal model; The obtaining of a multi-task training data set includes: Acquire a plurality of initial training videos; wherein the initial training videos include human behaviors; For each of the initial training videos, performing frame processing on the initial training video to obtain a training video frame sequence corresponding to the initial training video; For each of the initial training videos, the training video frame sequence corresponding to the initial training video is divided into a first training video frame subsequence and a second training video frame subsequence, and a video consisting of the first training video frame subsequence is determined as a target training video; wherein the second training video frame subsequence is located after the first training video frame subsequence; For each of the initial training videos, generating the intent and instruction corresponding to the initial training video based on the training video frame sequence corresponding to the initial training video; For each of the initial training videos, determining task information corresponding to the initial training video based on scene information corresponding to the initial training video; wherein the task information includes at least one task; For each of the initial training videos, construct a mapping relationship between a target training video corresponding to the initial training video and the intention, instruction, and task information corresponding to the initial training video to obtain the multi-task training data set; The generating the intent and instruction corresponding to the initial training video based on the training video frame sequence corresponding to the initial training video includes: Inputting a first initial training video frame of the training video frame sequence into a visual language model to obtain a first video subtitle of the first initial training video frame; wherein the visual language model is a pre-trained neural network model; Inputting the first video subtitle and a second initial training video frame of the training video frame sequence into the visual language model to obtain a second video subtitle of the second initial training video frame; Inputting the second video subtitle and the third initial training video frame of the training video frame sequence into the visual language model to obtain a third video subtitle of the third initial training video frame; Iterate the steps of inputting the first video subtitle and the second initial training video frame of the training video frame sequence into the visual language model to obtain the second video subtitle of the second initial training video frame until the video subtitle of the last initial training video frame of the training video frame sequence is obtained; Inputting each video subtitle into a preset large language model in sequence to obtain first language description information, and generating the instruction based on the first language description information; wherein the large language model is a pre-trained neural network model; The video subtitles corresponding to each initial training video frame in the second training video frame subsequence are sequentially input into the large language model to obtain second language description information, and the intent is generated based on the second language description information.

5. A terminal device, characterized in that: The terminal device includes a processor, a memory, and a computer program stored in the memory and executable by the processor, wherein when the computer program is executed by the processor, the multimodal model-based intent recognition method as described in any one of claims 1 to 3 is implemented.

6. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, wherein when the computer program is executed by a processor, the multimodal model-based intention recognition method as described in any one of claims 1 to 3 is implemented.

Images