Training and use of embodied perception localization model based on multimodal large model

Abstract

The present application relates to the field of visual positioning, and discloses a training and use method of an embodied perception positioning model based on a multimodal large model. The training method includes: obtaining general data and robot data; processing the general data and robot data into a unified task training format; obtaining text features through a text encoder of a large language model; obtaining corresponding image features through an extractor; obtaining feature representation of the feature space of the final large language model through a learnable projection layer; inputting the extracted text features and aligned image features to the large language model to generate text replies and calculating cross entropy loss with the data truth value. The present application solves the technical problems of high cost of expanding manually annotated data and inaccurate target detection and positioning in the prior art, and realizes that the objects detected and positioned by the model are richer, the environmental information is better perceived, and the complex text instructions are understood and reasoned, so as to cope with more complex perception and positioning tasks.

Description

Technical Field

[0001] The present application relates to the field of visual positioning, and in particular to the training and use methods of an embodied perception positioning model based on a multimodal large model. Background Art

[0002] In order to simulate human perception, robots are generally equipped with a variety of sensors. The multimodal information captured by these sensors, such as vision, depth, text, and sound, is integrated and processed to help robots better understand the surrounding environment and complete tasks. Among them, in the field of target object perception in visual modality perception, the task of target positioning is to find all the targets of interest in the image and determine their categories and positions. It is one of the core issues in the field of computer vision. Since various objects have different appearances, shapes, and postures, coupled with interference from factors such as illumination and occlusion during imaging, target detection has always been the most challenging problem in the field of computer vision. In order to improve the accuracy of target positioning, different types of target detection algorithms and models have emerged, such as traditional two-stage and one-stage algorithms, such as GLIP and Grounding Dino models.

[0003] At present, in vision development, some standard traditional perception models are often used for training to locate a fixed set of predefined target classes in crowd-labeled datasets. These datasets have fewer target classes, and these models cannot be generalized well to perception and positioning tasks in the open world. Moreover, the cost of expanding such manually labeled data is very high. In addition, the target positioning models currently built based on phrase positioning tasks, such as GLIP and Grounding Dino, lack strong text understanding capabilities in text instruction feature extraction due to the small language model used. When it comes to text instructions describing complex objects, they will not be accurately positioned due to insufficient understanding and reasoning capabilities. Summary of the invention

[0004] The embodiments of the present application solve the technical problems of high cost of expanding manually labeled data and inaccurate target detection and positioning in the prior art by providing a training and use method for an embodied perception and positioning model based on a multimodal large model, thereby achieving richer objects detected and positioned by the model, better perception of environmental information, and understanding and reasoning of complex text instructions, so as to cope with more complex perception and positioning tasks.

[0005] The embodiment of the present application provides a training method for an embodied perception positioning model based on a multimodal large model, comprising: obtaining general data and robot data; processing the general data and the robot data into a unified task training format; obtaining text features through a text encoder of a large language model; obtaining corresponding image features through an extractor; obtaining a feature representation of the feature space of the final large language model through a learnable projection layer; inputting the extracted text features and aligned image features to the large language model to generate a text reply and calculating the cross entropy loss with the data true value; wherein the location information of the object is in the form of pure natural language, i.e., numbers, and is extracted through a text encoder to obtain corresponding features.

[0006] Furthermore, the large language model is Flan t5xl, and the Lora method is used to fine-tune the large language model.

[0007] Furthermore, multi-task joint training is adopted in the training process, and the joint training tasks include basic visual description, region-level reference expression understanding and reference expression generation, multi-object detection, visual question answering data and relational reasoning data.

[0008] Furthermore, a phased training is adopted in the training process, including three stages of training: the first stage training includes the perception positioning task data and the visual description task of the general data; the second stage training includes the joint training of other tasks of the general data, and the introduction of the perception positioning data of the first stage for continued training, wherein the other tasks of the general data include multi-target detection, visual answering, and visual reasoning question and answering; the third stage training includes the introduction of robot data, and the perception positioning data and multi-target detection data of the general data are mixed with the robot perception task data, and the ratio of robot data to general data is adjusted to 8:2.

[0009] Furthermore, in the second stage of training, according to the total amount of training data in this stage and the amount of data in a set epoch, the proportion is adjusted so that the perception data of the first stage is maintained at 5%-10% of the total amount of data.

[0010] Furthermore, a multi-level visual feature extractor is used in the training process, and the image is input into the multi-level visual feature encoder in the feature extraction process of the image. In the image encoding process, the image feature samples extracted by different layers of visual encoders are fused to extract multi-level semantic and image information.

[0011] The present application also provides a method for using an embodied perception positioning model based on a multimodal large model, including: obtaining text instructions and observed images; passing the text instructions through a text feature extractor of a large language model to obtain corresponding text instruction features, and extracting the observed image through a multi-level visual feature extractor to obtain corresponding visual features; the visual features pass through a learnable projection layer to obtain a visual feature representation aligned with the feature space of the large language model; splicing the aligned visual features with the extracted text features to obtain a multimodal feature that is a fusion of environmental information and text instructions as input to the large language model; the large language model decodes the bounding box coordinates of the output object by autoregression.

[0012] One or more technical solutions provided in the embodiments of the present application have at least the following technical effects or advantages:

[0013] 1. Due to the phased training and the combination of general data and robot operation data, when robot data is scarce, the introduction of general data enables the trained embodied perception model to be better generalized and transferred to robot perception and positioning tasks.

[0014] 2. By introducing a multimodal large language model, we can better perceive environmental information and understand and reason about complex text instructions, so as to cope with more complex perception and positioning tasks.

[0015] 3. By unifying target detection and positioning into the phrase positioning task and formulating the position input / output in the most natural and flexible language form, namely numbers, the types of training data are expanded, making the objects detected and positioned by the model richer. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 It is a flowchart of a training method of an embodied perception positioning model based on a multimodal large model in an embodiment of the present application;

[0017] Figure 2 A schematic diagram of the process of extracting image features by a multi-level visual feature extractor in an embodiment of the present application;

[0018] Figure 3 This is a schematic diagram of multi-task joint training in an embodiment of the present application;

[0019] Figure 4 It is a flow chart of a method for using an embodied perception positioning model based on a multimodal large model in an embodiment of the present application;

[0020] Figure 5 This is a reasoning diagram of the embodied perception positioning model in the embodiment of the present application during use based on a multimodal large model. DETAILED DESCRIPTION

[0021] In order to make the basic method of the embodiments of the present application more obvious and easy to understand, the specific embodiments of the present application are described in detail below with reference to the accompanying drawings.

[0022] Embodiment 1

[0023] Figure 1 This is a training method for an embodied perception positioning model based on a multimodal large model in an embodiment of the present application, which is described in detail through specific steps below.

[0024] S1, obtain general data and robot data.

[0025] In specific implementations, general data can come from public detection datasets or large-scale phrase localization data on the Internet, such as RefCOCO, Visual Genome, etc., while robot data is data collected through the robot's own perception and operation, such as Open-X-Embodiment data, etc.

[0026] S2, processes all common data and robot data into a unified task training format.

[0027] In a specific implementation, the task training format may be similar to the following format:

[0028] Question: Please give the coordinates of object A in the picture.

[0029] answer: <bbox>

[0030] The position information of object A is expressed by the bounding box coordinate code, such as: the bounding box coordinate code is [x min ,y min ,x max ,y max ], x min and min Indicates the coordinates of the upper left corner of the border, x max and max Indicates the bottom right coordinate of the border.

[0031] S3, obtains text features through the text encoder of the large language model.

[0032] In a specific implementation, the large language model can be Flant5xl, and the Lora method can be used to fine-tune the large language model. During the fine-tuning process, targeted parameter updates are used to avoid over-adjustment, thereby maintaining the overall performance of the model in multi-task applications. At the same time, the Lora method also significantly improves the training speed of the model and reduces the demand for training resources by effectively utilizing previously learned knowledge.

[0033] S4, the image is extracted to obtain the corresponding image features.

[0034] In a specific implementation, a multi-level visual feature extractor can be used to extract the corresponding image features of the image. During the feature extraction process of the image, the image is input into a multi-level visual feature encoder. During the image encoding process, the image feature samples extracted by different layers of visual encoders are fused to extract multi-level semantics and image information.

[0035] In the specific implementation, Figure 2 As shown, a multi-level visual feature extractor is used to extract the corresponding image features of the image, and the fused feature z = w1LLN(z1)+…+w M LLN(z M ).

[0036] S5, after a learnable projection layer, obtains the feature representation of the final large language model feature space.

[0037] In a specific implementation, the location information of the object is in the form of pure natural language, i.e., numbers, and is extracted by a text encoder to obtain corresponding features.

[0038] S6, the extracted text features and aligned image features are input into the large language model to generate text responses and calculate the cross entropy loss with the true value of the data.

[0039] During the model training process, the following three methods can be used to enhance the perceptual positioning ability of the embodied perceptual positioning model.

[0040] The first one, refer to Figure 3 As shown in the figure, multi-task joint training is adopted in the model training process. The joint training tasks include basic visual description (Captioning), perceptual positioning (REC), reference expression generation (REG), multi-target detection, etc.

[0041] Visual description task (Caption): Given an image, generate a description of the image.

[0042] Perceptual localization (REC): Given a text description and an image, locate the position of the object described in the text in the image.

[0043] Reference Expression Generation (REG): Given an image and a location box, generate a specific description for the location region.

[0044] Multi-target detection: Given an image and a text description, locate the position of the described object in the image, where the image may contain multiple related objects.

[0045] The second method is to use phased training during model training, which includes three phases of training:

[0046] The first stage of training includes the perceptual localization task data of general data and the visual description task.

[0047] In the first stage of training, the learning rate can be set to 5e-4, the batch size is 8, and 10 epochs are trained on 4 NVIDIA RTX A100 (40GB) graphics cards.

[0048] The second stage of training includes joint training of other tasks on common data, while introducing the perception and positioning data from the first stage for continued training. Other tasks on common data include multi-target detection, visual answering, and visual reasoning question and answering.

[0049] Among them, in the second stage of training, the proportion can be adjusted according to the total amount of training data in this stage and the amount of data in an epoch set so that the perception data of the first stage remains at 5%-10% of the total data, and the perception task is continued to be trained to avoid a large loss of perception task capabilities during the training process.

[0050] The third stage of training includes introducing robot data, mixing the perception positioning data and multi-target detection data of general data with the robot perception task data, and adjusting the ratio of robot data to general data to 8:2, so that the robot data is maintained at a larger ratio, and continuing training to fine-tune the perception tasks to adapt to the robot field.

[0051] The training is conducted in stages, and general data is combined with robot operation data. When robot data is scarce, the introduction of general data enables the trained embodied perception model to be better generalized and transferred to robot perception and positioning tasks.

[0052] The third method uses a multi-level visual feature extractor during training. During the feature extraction process of the image, the image is input into the multi-level visual feature encoder. During the image encoding process, the image feature samples extracted by the visual encoders at different levels are fused to extract multi-level semantic and image information. Finally, the visual extractor model inputs the fused rich visual features into the learnable projection layer, aligns them with the instruction space of the large language model, and finally outputs them to the large model for predicting position information.

[0053] In summary, since the embodiment of the present application uses a multimodal large model for perception and positioning, by introducing a multimodal large language model, it can better perceive environmental information and understand and reason complex text instructions, so as to cope with more complex perception and positioning tasks. In addition, by unifying target detection and positioning into the phrase positioning task, and formulating the position input / output as the most natural and flexible language form, namely numbers, the types of training data are expanded, making the objects detected and positioned by the model more abundant.

[0054] In order to enable those skilled in the art to better understand and implement the embodiments of the present application, the following reference is made to Figure 4 A method for using an embodied perception positioning model based on a multimodal large model is introduced accordingly.

[0055] Embodiment 2

[0056] Reference Figure 4 As shown, the embodiment of the present application provides a method for using an embodied perception positioning model based on a multimodal large model, and the method for using the embodied perception positioning model includes:

[0057] Get text instructions and observation images.

[0058] The text instructions are passed through the text feature extractor of the large language model to obtain the corresponding text instruction features, and the observed image is extracted through the multi-level visual feature extractor to obtain the corresponding visual features.

[0059] The visual features are passed through a learnable projection layer to obtain visual feature representations that are aligned with the large language model feature space.

[0060] The aligned visual features and the extracted text features are concatenated to obtain multimodal features that fuse environmental information and text instructions, which serve as the input of the large language model.

[0061] The large language model decodes the bounding box coordinates of the output object through autoregression.

[0062] Among them, Figure 5 As shown in the figure, the coordinate information of the object output by the model is the value normalized according to the size of the processed image. By performing corresponding operations on the output according to the image size, the corresponding [x min ,y min ,x max ,y max ] is used to indicate .

[0063] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.

[0064] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A system that specifies the functions of a box or multiple boxes.

[0065] These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer-readable memory produce a product including an instruction system, which is implemented in the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0066] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps of the functions specified in a box or multiple boxes. Although the preferred embodiments of the present application have been described, those skilled in the art may make additional changes and modifications to these embodiments once they know the basic creative concept. Therefore, the attached claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the present application.

[0067] Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is also intended to include these modifications and variations.< / bbox>

Claims

1. A training method for an embodied perception positioning model based on a multimodal large model, characterized in that: include: Get general data and robot data; Processing the general data and the robot data into a unified task training format; Get text features through the text encoder of the large language model; The image is passed through the extractor to obtain the corresponding image features; After a learnable projection layer, the feature representation of the final large language model feature space is obtained; The extracted text features and aligned image features are fed into the large language model to generate text responses and the cross entropy loss is calculated with the true value of the data. Among them, the location information of the object is in the form of pure natural language, that is, numbers, and the corresponding features are extracted by the text encoder; The training process adopts phased training, including three phases of training: The first phase of training includes perceptual localization task data of general data and visual description task; The second phase of training includes joint training of other tasks on common data, while introducing the perception and positioning data from the first phase for continued training. The other tasks on the common data include multi-target detection, visual answering, and visual reasoning question answering. The third stage of training includes introducing robot data, mixing the perception positioning data and multi-target detection data of general data with the robot perception task data, and adjusting the ratio of robot data to general data to 8:2; In the second stage of training, the ratio is adjusted according to the total amount of training data in this stage and the amount of data in an epoch so that the perception data in the first stage remains at 5%-10% of the total data.

2. The method for training an embodied perception positioning model based on a multimodal large model as claimed in claim 1, characterized in that: The large language model is Flan t5xl, and the Lora method is used to fine-tune the large language model.

3. The training method of the embodied perception positioning model based on the multimodal large model as claimed in claim 1, characterized in that: Multi-task joint training is adopted during the training process. The joint training tasks include basic visual description, region-level reference expression understanding and reference expression generation, multi-object detection, visual question answering data and relational reasoning data.

4. The method for training an embodied perception positioning model based on a multimodal large model as claimed in claim 1, characterized in that: A multi-level visual feature extractor is used in the training process. In the feature extraction process of the image, the image is input into the multi-level visual feature encoder. In the image encoding process, the image feature samples extracted by the visual encoders at different levels are fused to extract multi-level semantic and image information.

Images