Visual task processing method and device, electronic equipment, medium and product

Abstract

The invention discloses a visual task processing method and device, electronic equipment, a medium and a product. The method comprises the following steps: acquiring a video image sequence and a user instruction; and inputting the video image sequence and the user instruction into a task preference model, expanding a new visual task through a task decoder and a multi-mode large language model in the task preference model, and outputting a visual task processing result. According to the method, the understanding capability of the multi-modal large language model on the visual task is enhanced by expanding a new visual task, so that the visual task can be perceived and understood more accurately.

Description

Technical Field

[0001] Embodiments of the present invention relate to the field of artificial intelligence technology, and in particular, to a visual task processing method, device, electronic device, medium and product. Background Art

[0002] The task of visual question answering is a fundamental problem in the field of artificial intelligence and has received increasing attention at the intersection of computer vision and natural language processing.

[0003] Existing technologies generally use multimodal large language models to perceive and reason about visual tasks, and obtain task results after execution. This often comes at the expense of the understanding performance of the multimodal large language model, and is unable to accurately perceive visual tasks, resulting in inaccurate output task results and incomplete content. Summary of the invention

[0004] The present invention provides a visual task processing method, device, electronic device, medium and product to solve the problem in the prior art that the accuracy of the output task results is low when a multimodal large language model is used to perform visual task processing.

[0005] According to one aspect of the present invention, a visual task processing method is provided, comprising:

[0006] Acquire video image sequences and user instructions;

[0007] The video image sequence and the user instruction are input into a task preference model, a new visual task is expanded through a task decoder and a multimodal large language model in the task preference model, and a visual task processing result is output.

[0008] According to another aspect of the present invention, there is provided a visual task processing device, comprising:

[0009] An acquisition module, used for acquiring video image sequences and user instructions;

[0010] The execution module is used to input the video image sequence and the user instruction into a task preference model, expand a new visual task through a task decoder and a multimodal large language model in the task preference model, and output a visual task processing result.

[0011] According to another aspect of the present invention, an electronic device is provided, the electronic device comprising:

[0012] at least one processor;

[0013] and a memory communicatively coupled to the at least one processor;

[0014] The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can perform the visual task processing method described in any embodiment of the present invention.

[0015] According to another aspect of the present invention, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the visual task processing method described in any embodiment of the present invention when executed.

[0016] According to another aspect of the present invention, a computer program product is provided. The computer program product includes a computer program. When the computer program is executed by a processor, the visual task processing method described in any embodiment of the present invention is implemented.

[0017] The technical solution of the embodiment of the present invention expands new visual tasks through the task decoder and the multimodal large language model in the task preference model, thereby achieving the beneficial effect of more accurate perception and understanding of visual tasks.

[0018] It should be understood that the contents described in this section are not intended to identify the key or important features of the embodiments of the present invention, nor are they intended to limit the scope of the present invention. Other features of the present invention will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0020] Figure 1 A flowchart of a visual task processing method provided in Embodiment 1 of the present invention;

[0021] Figure 2 A flowchart of a visual task processing method provided in Embodiment 2 of the present invention;

[0022] Figure 3 A schematic diagram of a specific embodiment of the present invention;

[0023] Figure 4 A comparison chart of multimodal video understanding capabilities of different models provided by an embodiment of the present invention;

[0024] Figure 5 A comparison chart of the video time-sequence clue question-answering capabilities of different models provided by an embodiment of the present invention;

[0025] Figure 6 A comparison chart of the image understanding capabilities of different models provided by an embodiment of the present invention;

[0026] Figure 7 Zero-sample performance of different models for time series positioning provided by embodiments of the present invention;

[0027] Figure 8 A comparison chart of the timing positioning and wonderful moment editing fine-tuning performance of different models provided in an embodiment of the present invention;

[0028] Fig. 9 A comparison chart of the performance of different models for spatial positioning tasks provided by an embodiment of the present invention;

[0029] Fig.10 A comparison chart of the tracking task performance of different models provided in an embodiment of the present invention;

[0030] Fig.11 A comparison diagram of segmentation performance of different models provided in an embodiment of the present invention;

[0031] Fig.12 A schematic diagram of the structure of a visual task processing device provided in Embodiment 3 of the present invention;

[0032] Fig.13 A schematic diagram of the structure of an electronic device for a visual task processing method according to an embodiment of the present invention. DETAILED DESCRIPTION

[0033] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiment of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiment of the present invention. Obviously, the described embodiment is only an embodiment of a part of the present invention, rather than all the embodiments. Based on the embodiments in the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work should belong to the scope of protection of the present invention. It should be understood that the various steps recorded in the method implementation of the present invention can be performed in different orders and / or in parallel. In addition, the method implementation may include additional steps and / or omit the steps shown. The scope of the present invention is not limited in this respect.

[0034] The term "including" and its variations used herein are open inclusions, i.e., "including but not limited to". The term "based on" means "based at least in part on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments". The relevant definitions of other terms will be given in the following description.

[0035] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0036] It should be noted that the modifications of "one" and "plurality" mentioned in the present invention are illustrative rather than restrictive, and those skilled in the art should understand that, unless otherwise clearly indicated in the context, it should be understood as "one or more".

[0037] The names of the messages or information exchanged between multiple devices in the embodiments of the present invention are only used for illustrative purposes, and are not used to limit the scope of these messages or information.

[0038] Embodiment 1

[0039] Figure 1 A flowchart of a visual task processing method provided in Embodiment 1 of the present invention is provided. The method can be applicable to specific visual task processing situations. The method can be performed by a visual task processing device, wherein the device can be implemented by software and / or hardware and is generally integrated on an electronic device. In this embodiment, the electronic device includes but is not limited to: a computer device.

[0040] like Figure 1 As shown, a visual task processing method provided by Embodiment 1 of the present invention includes the following steps:

[0041] S110 , obtaining a video image sequence and a user instruction.

[0042] Among them, the video image sequence is a collection of image frames arranged in time order. There is a strong temporal correlation between adjacent frames in the video image sequence, and there is also spatial correlation within each frame of the image itself.

[0043] In this embodiment, the video image sequence can be obtained in a variety of ways, such as a user shooting a video and then transmitting the video to an electronic device; a user can also use a camera of an electronic device to shoot a video to obtain a video image sequence; a previously shot video image sequence can also be obtained from a storage device or optical disc; and a video image sequence can also be downloaded from a video library.

[0044] Among them, user instructions refer to the specific requirements, operation instructions or inquiries issued by the user to the electronic device.

[0045] In this embodiment, user instructions can be obtained in a variety of ways, such as a user inputting text instructions through a keyboard in an electronic device so that the electronic device obtains the text instructions; the user can also issue them through voice, and the electronic device obtains voice instructions through a smart speaker or voice assistant.

[0046] S120, inputting the video image sequence and the user instruction into a task preference model, expanding a new visual task through a task decoder and a multimodal large language model in the task preference model, and outputting a visual task processing result.

[0047] In this embodiment, a learnable task decoder is introduced to construct a task preference model, receive the output from the multimodal large language model, and dynamically activate the corresponding task tag and task decoder according to the user's instructions to process the visual task and output the visual task processing result. Among them, new visual tasks can be expanded through the task decoder to process different visual tasks.

[0048] In this embodiment, the visual task processing result can be an answer to the user's instruction, and the visual task processing result is a text composed of multiple common task tokens and multiple special task tokens. In natural language processing, a token is the basic unit of text processing, usually referring to a word, punctuation mark or other meaningful language unit. For example, in the sentence "I like reading.", "I", "like", "reading", "." are all tokens. Among them, common task tokens can be text and punctuation marks, and special tokens can be time frames, space frames and mask data.

[0049] In this embodiment, the task preference model includes a multimodal large language model and multiple task decoders, and the multiple task decoders have one-to-one corresponding task tags.

[0050] Among them, the task preference model is obtained by adding multiple task decoders to a multimodal large language model for model training. A task decoder corresponds to processing a specific visual task. Each task decoder has a corresponding task tag. Based on the task tag, the corresponding visual information features can be generated by the multimodal large language model. The corresponding visual information features are input into the task decoder corresponding to the task tag for decoding to output a special task token.

[0051] A visual task processing method provided in a first embodiment of the present invention first obtains a video image sequence and a user instruction; then the video image sequence and the user instruction are input into a task preference model, a new visual task is expanded through a task decoder and a multimodal large language model in the task preference model, and a visual task processing result is output. On the premise of improving multimodal performance, the above method uses traditional visual tasks in an extensible manner to enhance the multimodal large language model's ability to understand visual tasks, so that it can perceive and understand visual content more accurately and achieve better performance in various visual tasks.

[0052] Furthermore, the training process of the task preference model includes:

[0053] Using multiple dialogue templates to train the multimodal large language model on common visual tasks;

[0054] Integrate task tags and task decoders into the multimodal large language model, and use multiple task data for special visual task training;

[0055] Perform multi-task training using the dialogue template and the multiple task data;

[0056] The above training process is performed until the difference between the model's loss value and the target value is less than the preset value, and the task preference model is obtained;

[0057] The loss value of the model includes the loss value of a multimodal large language model, the loss value of a general visual task training, and the loss value of a plurality of special visual task trainings.

[0058] In this embodiment, the task preference model adopts a three-stage local-to-global training scheme to gradually integrate visual task knowledge into the multimodal large language model, and uses multi-task collaborative training to enhance the generalization ability of the task preference model.

[0059] Among them, model training includes three stages. Stage 1 is general visual task training, which is used to train the model to generate the accuracy of general task tokens; Stage 2 is multiple special visual task training, which is used to train the model to generate special task tokens and the accuracy of task output; Stage 3 includes visual encoders, visual language connectors, multimodal large language models and multiple task decoders, all of which will participate in the training of the model.

[0060] Among them, the loss value of the multimodal large language model will be used in each training stage, the loss value of the general visual task training will be used in both stage 1 and stage 3; the loss value of multiple special visual task training is the sum of the loss values ​​of multiple task decoders, which will be used in both stage 2 and stage 3.

[0061] Furthermore, a dialogue template includes a video image sequence, a user instruction, and a common task label, and the using of multiple dialogue templates to perform common task training on the multimodal large language model includes:

[0062] Multiple video image sequences, multiple user instructions and multiple real task labels are input into the multimodal large language model; for the one dialogue template, the user intention is identified by the low-rank adaptive fine-tuning module of the multimodal large language model to generate a common task token; based on the one common task token and the corresponding common task label, the loss value of the common visual task training is calculated.

[0063] Furthermore, a task data includes a video image sequence, a user instruction and a special task label, and the task label and the task decoder are integrated into the multimodal large language model, and a plurality of task data are used for special visual task training, including:

[0064] Input multiple video image sequences, multiple user instructions and multiple real task labels into the multimodal large language model; for the one task data, generate a special task token through the low-rank adaptive fine-tuning module of the multimodal large language model; replace the one special task token with a task tag, generate information features through the low-rank adaptive fine-tuning module, and send the information features to the corresponding task decoder to obtain an output result; use the loss function to calculate a loss value of a special visual task training using the output result and the real special task label; calculate the sum of the loss values ​​of multiple special visual task trainings to obtain the loss values ​​of multiple special visual task trainings.

[0065] Optionally, the multiple task decoders may be already trained, or the multiple task decoders may be randomly initialized.

[0066] Embodiment 2

[0067] Figure 2 This is a flow chart of a visual task processing method provided by Embodiment 2 of the present invention. Embodiment 2 is optimized on the basis of the above embodiments. For details not yet fully described in this embodiment, please refer to the above embodiments.

[0068] like Figure 2 As shown, a visual task processing method provided by Embodiment 2 of the present invention includes the following steps:

[0069] S210: Acquire a video image sequence and a user instruction.

[0070] S220: Input the video image sequence and the user instruction into a task preference model, and generate a first common task token based on the user instruction and the video image sequence through the multimodal large language model.

[0071] In this embodiment, the multimodal large language model can perform semantic recognition on user instructions, understand the user's intention, and generate common task tokens based on the video image sequence.

[0072] S230. Input the first common task token, the user instruction, and the video image sequence into the task preference model, and after determining to generate a common task token through the multimodal large language model, generate a second common task token based on the first common task token, the user instruction, and the video image sequence.

[0073] Among them, after model training, the multimodal large language model can determine whether to generate a normal task token or a special task token based on the input content. During the training process, the multimodal large language model can determine whether to generate a normal task token or a special task token by performing semantic recognition on the input user instructions and normal task tokens.

[0074] In this embodiment, the multimodal large language model can perform semantic recognition on the user instructions and the first common task token, and after confirming the output of the common task token, the common task token can be generated according to the video image sequence.

[0075] S240, input the first ordinary task token, the second ordinary task token, the user instruction and the video image sequence into the task preference model, determine to generate a special token through the multimodal large language model, generate a special task token according to the first ordinary task token, the second ordinary task token, the user instruction and the video image sequence, call the corresponding task tag and task decoder according to the special task token, and output the special task token execution result based on the task decoder.

[0076] Furthermore, the task preference model also includes a visual encoder and a visual language connector, wherein the first common task token, the second common task token, the user instruction and the video image sequence are input into the task preference model, and after the special token is determined to be generated by the multimodal large language model, a special task token is generated according to the first common task token, the second common task token, the user instruction and the video image sequence, and a corresponding task tag and task decoder are called according to the special task token, and the special task token execution result is output based on the task decoder, including:

[0077] The first common task token, the second common task token, the video image sequence and the user instruction are input into the task preference model. After the multimodal large language model determines to generate a special token, the visual encoder is used to extract features of the video image sequence to obtain visual features; the visual language connector is used to map the visual features to the text space to obtain a visual feature vector; a special task token is generated according to the first common task token, the second common task token, the user instruction and the visual feature vector through the multimodal large language model; a corresponding task tag is called according to the special task token, and a corresponding visual information feature is generated according to the user input, the visual feature vector and the task tag through the multimodal large language model; the visual information feature is input into the corresponding task decoder for decoding and outputting the execution result of the special task token.

[0078] Among them, the visual information features may include temporal information features and fine-grained visual information features.

[0079] Among them, if the special task token is a time frame, the execution result of the special task token can be a time value; if the special task token is a space frame, the execution result of the special task token can be a space coordinate; if the special task token is mask data, the execution result of the special task token can be the real data corresponding to the mask data.

[0080] S250. Input the first common task token, the second common task token, the execution result of the special task token, the user instruction and the video image sequence into the task preference model, and repeat the above process until the termination character is generated, to obtain the text formed by the output of all common task tokens and the execution results of all special task tokens.

[0081] Among them, if the input task token includes ordinary task tokens and special task tokens, and after the ordinary task token is determined to be generated through the multimodal large language model, the next ordinary task token can be generated based on the input ordinary task token, user instructions and video image sequence.

[0082] Among them, the termination character is preset during the model training process, and the specific form of the termination character is not limited.

[0083] The second embodiment of the present invention provides a visual task processing method, which specifies the process of generating visual task processing results by a task preference model. This method can improve the overall multimodal performance while improving the understanding performance of specific tasks; multiple task decoders using task preference models can adapt to a wider range of visual tasks; and can significantly improve the performance of multimodal large language models in various visual tasks, including zero-sample learning capabilities.

[0084] The embodiment of the present invention provides a specific implementation method based on the technical solutions of the above embodiments.

[0085] As a specific implementation method of this embodiment, Figure 3 A flowchart of a specific embodiment provided for an embodiment of the present invention is as follows: Figure 3 As shown, the following process is included:

[0086] Input the visual input (i.e., video image sequence) and user input (i.e., user instructions) into the task preference model. The visual input and user input generate a general task token "video" through the multimodal large language model. Input the visual input, user input, and "video" into the task preference model. The input content generates a general task token "display" through the multimodal large language model. Input "video", "display", and the visual input and user input into the task preference model, and generate a general task token "," through the multimodal large language model. Repeat the above process to obtain the general task tokens "that", "coordinates", and "is". Input all the above general task tokens, user input, and video input into the task preference model. After determining the generation of special tokens through the multimodal large language model, the visual input is subjected to feature extraction by the visual encoder to output visual features. The visual features are mapped to the text space through the visual language connector to obtain visual feature vectors. The special task token generated by the multimodal large language model based on all the above general task tokens, user instructions, and visual feature vectors is the spatial box. Call the corresponding task label v3 according to the special task token, and generate the corresponding visual information feature e3 through the multimodal large language model based on the user input, visual feature vectors, and task label v3. Input the visual information feature e3 into the task head 3 (i.e., task encoder 3) to output the special task token execution result as [98, 201, 135, 128]. Input the user input, visual input, all the generated general task tokens, and special task tokens into the task preference model, and generate a general task token "of" through the multimodal large language model based on the user input, visual input, and all the generated general task tokens. Repeat the above process, and the generated multiple general task tokens include "rabbit", "from", "is", "vegetable", "surrounded", "by", "fence", "closer", ".".”, “in”; input all the common task tokens and special task tokens generated above into the task preference model, determine the generation of special tokens through the multimodal large language model, extract features from the visual input through the visual encoder to output visual features, and map the visual features to the text space through the visual language connector to obtain a visual feature vector; the special task token generated by the multimodal large language model according to all the common task tokens, user instructions and visual feature vectors is used as a time frame, and the corresponding task tag v2 is called according to the special task token, and the corresponding visual information feature e2 is generated according to the user input, the visual feature vector and the task tag v2 through the multimodal large language model; the visual information feature e2 is input into the task head 2 (i.e., the task encoder 2) Output the special task token execution result as 9.4; input all the generated common task tokens and special task tokens into the task preference model to generate the common task token "seconds", and repeat the above process to generate the common task token "to"; input all the above generated common task tokens and special task tokens into the task preference model, and after determining the generation of special tokens through the multimodal large language model, extract the visual input through the visual encoder to output the visual features, and map the visual features to the text space through the visual language connector to obtain the visual feature vector; the special task token generated by the multimodal large language model according to all the above common task tokens, user instructions and visual feature vectors is used as the time frame, and the corresponding task tag is called according to the special task token v2, and generate the corresponding visual information feature e2 according to the user input, the visual feature vector and the task tag v2 through the multimodal large language model; input the visual information feature e2 into the task head 2 (i.e., the task encoder 2) to output the special task token and the execution result is 10.3; input all the generated common task tokens and special task tokens into the task preference model to generate the common task token "seconds", and repeat the above process to generate the common task tokens "between", ", "; input all the common task tokens and special task tokens generated above into the task preference model, and after determining the generation of special tokens through the multimodal large language model, the visual input is subjected to feature extraction through the visual encoder to output the visual features, and the visual features are mapped to the text through the visual language connector space to obtain a visual feature vector; use the multimodal large language model to generate a special task token as mask data based on all the above common task tokens, user instructions and the visual feature vector, call the corresponding task tag v1 according to the special task token, and use the multimodal large language model to generate the corresponding visual information feature e1 according to the user input, the visual feature vector and the task tag v1; input the visual information feature e1 into the task head 1 (i.e., task encoder 1) to output the special task token execution result as "the rabbit"; input all the generated common task tokens and special task tokens into the task preference model to generate the common task token "from", repeat the above process, and generate multiple common task tokens including "it", "above", "skip", ".” “Now”; input all the common task tokens and special task tokens generated above into the task preference model, and after determining the generation of special tokens through the multimodal large language model, extract features from the visual input through the visual encoder to output visual features, and map the visual features to the text space through the visual language connector to obtain a visual feature vector; the special task token generated by the multimodal large language model according to all the common task tokens, user instructions and visual feature vectors is used as mask data, and the corresponding task tag v1 is called according to the special task token, and the corresponding visual information feature e1 is generated according to the user input, the visual feature vector and the task tag v1 through the multimodal large language model; the visual information feature e1 is input into the task head 1 (i.e., task encoder 1) to output the special task token execution result as “the rabbit”; all the generated common task tokens and special task tokens are input into the task preference model to generate a common task token “than”, and the above process is repeated to generate multiple common task tokens including “it”, “from”, “wall”, and “closer”. ”; The termination character is recognized, the model input is stopped, and the output text content is "The video shows that the rabbit with coordinates [98, 201, 135, 128] is closer to the fence surrounded by vegetables. Between 9.4 seconds and 10.3 seconds, another rabbit jumped over it. Now the other rabbit is closer to the fence than it is.".

[0087] It should be noted that stage 1 in the figure represents general visual task training, stage 2 represents special visual task training, and stage 3 is multi-task training. Among them, the multimodal large language model participates in the model training of stage 1; the multimodal large language model, task label v i and multiple task decoders participate in model training in stage 2; visual encoder, visual connector, multimodal large language model, task tag v i And multiple task decoders participate in the model training of stage 3.

[0088] The present invention has conducted a number of simulation experiments, and the results show that a visual task processing method provided by an embodiment of the present invention can significantly improve the performance of a multimodal large language model. For example, it has an average improvement of 14.6% on multiple evaluation data sets, and has achieved performance comparable to the best existing models in tasks such as spatial positioning, time retrieval, highlight detection, reference segmentation and tracking.

[0089] Figure 4 A comparison chart of the multimodal video understanding capabilities of different models provided by the embodiment of the present invention, such as Figure 4 As shown in the figure, w / os. means no subtitles, ws. means subtitles, M-AVG refers to the average value of MLVU, and VideoChat-TPO refers to the task preference model. Figure 4From the data comparison, it can be seen that compared with other models, VideoChat-TPO has better multimodal video understanding ability.

[0090] Figure 5 A comparison chart of the video timing clue question-answering capabilities of different models provided by the embodiment of the present invention, such as Figure 5 As shown, the results show that compared with other models, VideoChat-TPO has better video timing clue question answering ability.

[0091] Figure 6 A comparison chart of the image understanding capabilities of different models provided by the embodiment of the present invention, such as Figure 6 As shown, the results show that VideoChat-TPO has better image understanding ability compared with the other 8 models.

[0092] Figure 7 The zero-sample performance of different models provided by the embodiments of the present invention for time series positioning, such as Figure 7 As shown, the model represented by gray font does not involve the large language model. The results show that compared with other models, VideoChat-TPO has better zero-sample performance for temporal localization.

[0093] Figure 8 A comparison chart of the timing positioning and wonderful moment editing fine-tuning performance of different models provided in an embodiment of the present invention, such as Figure 8 As shown, the model represented by gray font does not involve the large language model. The results show that compared with other models, VideoChat-TPO has better timing positioning and wonderful moment editing fine-tuning performance.

[0094] Fig. 9 A comparison chart of the performance of different models for spatial positioning tasks provided by the embodiments of the present invention, such as Fig. 9 As shown, the models marked with an asterisk include a task decoder. The results show that the model including the task decoder has better spatial positioning task performance than other models.

[0095] Fig.10 A comparison chart of the tracking task performance of different models provided in the embodiment of the present invention, such as Fig.10 As shown, the model including the task decoder has better tracking task performance than other models.

[0096] Fig.11 The comparison diagram of segmentation performance of different models provided in the embodiment of the present invention is as follows: Fig.11 As shown in the figure, VideoChat-TPO is a zero-sample test, and other models are tested after fine-tuning. The results show that compared with other models, VideoChat-TPO has better reference segmentation performance.

[0097] Embodiment 3

[0098] Fig.12 This is a schematic diagram of the structure of a visual task processing device provided in Embodiment 3 of the present invention. The device can be applicable to situations where specific visual tasks are processed. The device can be implemented by software and / or hardware and is generally integrated on an electronic device.

[0099] like Fig.12 As shown, the device includes: an acquisition module 110 and an execution module 120.

[0100] An acquisition module 110, used to acquire a video image sequence and a user instruction;

[0101] The execution module 120 is used to input the video image sequence and the user instruction into the task preference model, expand the new visual task through the task decoder and the multimodal large language model in the task preference model, and output the visual task processing result.

[0102] In this embodiment, the device first acquires a video image sequence and a user instruction through an acquisition module 110; then inputs the video image sequence and the user instruction into a task preference model through an execution module 120, expands a new visual task through a task decoder and a multimodal large language model in the task preference model, and outputs a visual task processing result.

[0103] This embodiment provides a visual task processing device that can perceive and understand visual tasks more accurately.

[0104] Furthermore, the task preference model includes a multimodal large language model and multiple task decoders, and the multiple task decoders have one-to-one corresponding task labels.

[0105] Furthermore, the execution module 120 includes:

[0106] A first generating unit, configured to input the video image sequence and the user instruction into a task preference model, and generate a first common task token based on the user instruction and the video image sequence through the multimodal large language model;

[0107] a second generating unit, configured to input the first common task token, the user instruction, and the video image sequence into a task preference model, and after determining to generate a common task token through the multimodal large language model, generate a second common task token according to the first common task token, the user instruction, and the video image sequence;

[0108] an output unit, configured to input the first common task token, the second common task token, the user instruction, and the video image sequence into a task preference model, and after determining to generate a special task token through the multimodal large language model, generate a special task token according to the first common task token, the second common task token, the user instruction, and the video image sequence, and call a corresponding task tag and a task decoder according to the special task token, and output a special task token execution result based on the task decoder;

[0109] The first common task token, the second common task token, the execution result of the special task token, the user instruction and the video image sequence are input into the task preference model, and the above process is repeated until the termination character is generated, so as to obtain the text formed by the output of all common task tokens and the execution results of all special task tokens.

[0110] On the basis of the above optimization, the task preference model also includes a visual encoder and a visual language connector, and the output unit is specifically used to: input the first ordinary task token, the second ordinary task token, the video image sequence and the user instruction into the task preference model, and after determining to generate a special token through the multimodal large language model, perform feature extraction on the video image sequence through the visual encoder to obtain visual features; map the visual features to the text space through the visual language connector to obtain a visual feature vector; generate a special task token according to the first ordinary task token, the second ordinary task token, the user instruction and the visual feature vector through the multimodal large language model; call the corresponding task tag according to the special task token, and generate the corresponding visual information feature according to the user input, the visual feature vector and the task tag through the multimodal large language model; input the visual information feature into the corresponding task decoder for decoding and outputting the execution result of the special task token.

[0111] The device also includes a training module for executing a training process of a task preference model.

[0112] Based on the above technical solution, the training module includes:

[0113] A first training unit, configured to perform common visual task training on the multimodal large language model using a plurality of dialogue templates;

[0114] A second training unit is used to integrate the task label and the task decoder into the multimodal large language model, and use multiple task data to perform special visual task training;

[0115] A third training unit, configured to perform multi-task training using the dialogue template and the plurality of task data;

[0116] The above training process is performed until the difference between the model's loss value and the target value is less than the preset value, and the task preference model is obtained.

[0117] The loss value of the model includes the loss value of a multimodal large language model, the loss value of a general visual task training, and the loss value of a plurality of special visual task trainings.

[0118] Furthermore, a dialogue template includes a video image sequence, a user instruction, and a common task label, and the first training unit is specifically used for:

[0119] Multiple video image sequences, multiple user instructions and multiple real task labels are input into the multimodal large language model; for the one dialogue template, the user intention is identified by the low-rank adaptive fine-tuning module of the multimodal large language model to generate a common task token; based on the one common task token and the corresponding common task label, the loss value of the common visual task training is calculated.

[0120] Furthermore, a task data includes a video image sequence, a user instruction and a special task label, and the second training unit is specifically used for:

[0121] Input multiple video image sequences, multiple user instructions and multiple real task labels into the multimodal large language model; for the one task data, generate a special task token through the low-rank adaptive fine-tuning module of the multimodal large language model; replace the one special task token with a task tag, generate information features through the low-rank adaptive fine-tuning module, and send the information features to the corresponding task decoder to obtain an output result; use the loss function to calculate a loss value of a special visual task training using the output result and the real special task label; calculate the sum of the loss values ​​of multiple special visual task trainings to obtain the loss values ​​of multiple special visual task trainings.

[0122] The above-mentioned visual task processing device can execute the visual task processing method provided by any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the execution method.

[0123] Embodiment 4

[0124] Figure 5A schematic diagram of the structure of an electronic device 10 that can be used to implement an embodiment of the present invention is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices (such as helmets, glasses, watches, etc.) and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present invention described and / or required herein.

[0125] like Figure 5 As shown, the electronic device 10 includes at least one processor 11, and a memory connected to the at least one processor 11, such as a read-only memory (ROM) 12, a random access memory (RAM) 13, etc., wherein the memory stores a computer program that can be executed by at least one processor, and the processor 11 can perform various appropriate actions and processes according to the computer program stored in the read-only memory (ROM) 12 or the computer program loaded from the storage unit 18 to the random access memory (RAM) 13. In the RAM 13, various programs and data required for the operation of the electronic device 10 can also be stored. The processor 11, the ROM 12, and the RAM 13 are connected to each other through a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0126] A number of components in the electronic device 10 are connected to the I / O interface 15, including: an input unit 16, such as a keyboard, a mouse, etc.; an output unit 17, such as various types of displays, speakers, etc.; a storage unit 18, such as a disk, an optical disk, etc.; and a communication unit 19, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 19 allows the electronic device 10 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0127] The processor 11 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The processor 11 performs the various methods and processes described above, such as a visual task processing method.

[0128] In some embodiments, the visual task processing method may be implemented as a computer program, which is tangibly contained in a computer-readable storage medium, such as a storage unit 18. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 10 via the ROM 12 and / or the communication unit 19. When the computer program is loaded into the RAM 13 and executed by the processor 11, one or more steps in the visual task processing method described above may be performed. Alternatively, in other embodiments, the processor 11 may be configured to perform the visual task processing method in any other appropriate manner (e.g., by means of firmware).

[0129] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0130] In some embodiments, the visual task processing method can be implemented as a computer program, which is invisibly included in a computer program product. The computer program implements the visual task processing method of the present invention when executed by a processor. The computer program product can be understood as a software product that mainly implements its solution through a computer program. The computer program for implementing the method of the present invention can be written in any combination of one or more programming languages. These computer programs can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that when the computer program is executed by the processor, the functions / operations specified in the flowchart and / or block diagram are implemented. The computer program can be executed entirely on the machine, partially on the machine, as an independent software package, partially on the machine and partially on a remote machine, or entirely on a remote machine or server.

[0131] In the context of the present invention, a computer-readable storage medium may be a tangible medium that may contain or store a computer program for use by or in combination with an instruction execution system, device or equipment. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices or equipment, or any suitable combination of the foregoing. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0132] To provide interaction with a user, the systems and techniques described herein may be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and a pointing device (e.g., a mouse or trackball) through which the user can provide input to the electronic device. Other types of devices may also be used to provide interaction with the user; for example, the feedback provided to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form (including acoustic input, voice input, or tactile input).

[0133] The systems and techniques described herein may be implemented in a computing system that includes backend components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes frontend components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such backend components, middleware components, or frontend components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), a blockchain network, and the Internet.

[0134] A computing system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The client and server relationship is generated by computer programs running on the corresponding computers and having a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and VPS services.

[0135] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps described in the present invention can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solution of the present invention can be achieved, and this document does not limit this.

[0136] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A visual task processing method, characterized in that: The method comprises: Acquire video image sequences and user instructions; The video image sequence and the user instruction are input into a task preference model, a new visual task is expanded through a task decoder and a multimodal large language model in the task preference model, and a visual task processing result is output.

2. The method according to claim 1, characterized in that The task preference model includes a multimodal large language model and multiple task decoders, wherein the multiple task decoders have one-to-one corresponding task labels.

3. The method according to claim 1, characterized in that The step of inputting the video image sequence and the user instruction into a task preference model, expanding a new visual task through a task decoder and a multimodal large language model in the task preference model, and outputting a visual task processing result includes: Inputting the video image sequence and the user instruction into a task preference model, and generating a first common task token based on the user instruction and the video image sequence through the multimodal large language model; Inputting the first common task token, the user instruction, and the video image sequence into a task preference model, and after determining to generate a common task token through the multimodal large language model, generating a second common task token according to the first common task token, the user instruction, and the video image sequence; Inputting the first common task token, the second common task token, the user instruction and the video image sequence into the task preference model, and after determining to generate a special task token through the multimodal large language model, generating a special task token according to the first common task token, the second common task token, the user instruction and the video image sequence, calling a corresponding task tag and a task decoder according to the special task token, and outputting a special task token execution result based on the task decoder; The first common task token, the second common task token, the execution result of the special task token, the user instruction and the video image sequence are input into the task preference model, and the above process is repeated until the termination character is generated, so as to obtain the text formed by the output of all common task tokens and the execution results of all special task tokens.

4. The method according to claim 3, characterized in that The task preference model also includes a visual encoder and a visual-language connector, The first common task token, the second common task token, the user instruction and the video image sequence are input into the task preference model, and after determining to generate a special task token through the multimodal large language model, a special task token is generated according to the first common task token, the second common task token, the user instruction and the video image sequence, and a corresponding task tag and task decoder are called according to the special task token, and a special task token execution result is output based on the task decoder, including: The first common task token, the second common task token, the video image sequence and the user instruction are input into the task preference model, and after determining to generate a special token through the multimodal large language model, the video image sequence is subjected to feature extraction through the visual encoder to obtain visual features; Mapping the visual feature to the text space through the visual language connector to obtain a visual feature vector; Generate a special task token according to the first common task token, the second common task token, the user instruction and the visual feature vector through a multimodal large language model; Calling a corresponding task tag according to the special task token, and generating a corresponding visual information feature according to the user input, the visual feature vector and the task tag through the multimodal large language model; The visual information features are input into the corresponding task decoder for decoding and outputting the special task token execution result.

5. The method according to claim 1, characterized in that The training process of the task preference model includes: Using multiple dialogue templates to train the multimodal large language model on common visual tasks; Integrate task tags and task decoders into the multimodal large language model, and use multiple task data for special visual task training; Perform multi-task training using the dialogue template and the multiple task data; The above training process is performed until the difference between the model's loss value and the target value is less than the preset value, and the task preference model is obtained; The loss value of the model includes the loss value of a multimodal large language model, the loss value of a general visual task training, and the loss value of a plurality of special visual task trainings.

6. The method according to claim 5, characterized in that A dialogue template consists of a video image sequence, a user instruction, and a common task label. The using multiple dialogue templates to perform common task training on the multimodal large language model includes: Inputting multiple video image sequences, multiple user instructions, and multiple real task labels into the multimodal large language model; For the one dialogue template, identifying the user intention by using the low-rank adaptive fine-tuning module of the multimodal large language model to generate a common task token; A loss value of common visual task training is calculated based on the common task token and the corresponding common task label.

7. The method according to claim 5, characterized in that A task data consists of a video image sequence, a user instruction and a special task label. The task marker and the task decoder are integrated into the multimodal large language model, and a plurality of task data are used for special visual task training, including: Inputting multiple video image sequences, multiple user instructions, and multiple real task labels into the multimodal large language model; For the one task data, generating a special task token through a low-rank adaptive fine-tuning module of the multimodal large language model; Replacing the special task token with a task tag, generating information features through the low-rank adaptive fine-tuning module, and sending the information features to the corresponding task decoder to obtain an output result; Using the loss function to calculate a loss value for special visual task training, the output result and the actual special task label; The sum of the loss values ​​of multiple special visual task trainings is calculated to obtain the loss values ​​of multiple special visual task trainings.

8. A visual task processing device, characterized in that: The device comprises: An acquisition module, used for acquiring video image sequences and user instructions; The execution module is used to input the video image sequence and the user instruction into a task preference model, expand a new visual task through a task decoder and a multimodal large language model in the task preference model, and output a visual task processing result.

9. An electronic device, characterized in that: The electronic device comprises: at least one processor; and a memory communicatively coupled to the at least one processor; The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the visual task processing method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the visual task processing method according to any one of claims 1 to 7 when executed.

11. A computer program product, characterized in that The computer program product comprises a computer program, which, when executed by a processor, implements the visual task processing method according to any one of claims 1 to 7.

Images