Weight estimation method based on visual large model

Abstract

The invention discloses a weight estimation method based on a visual large model, and the method comprises the steps: obtaining a multi-modal weight data set, constructing a visual language model, and carrying out the training of two stages of the visual language model based on the multi-modal weight data set; obtaining a fine tuning data set based on the multi-modal weight data set; obtaining a weight estimation model for the visual language model completing the second-stage training based on the fine tuning data set, a category guidance strategy and a reference calibration strategy; and performing weight estimation based on the weight estimation model and a to-be-evaluated object image. According to the method, the weight estimation precision and the weight reasoning capability of the large visual model are effectively improved through the category guide strategy and the reference calibration strategy.

Description

Technical Field

[0001] The invention belongs to the technical field of machine vision, and in particular relates to a weight estimation method based on a large visual model. Background Art

[0002] With the rapid development of artificial intelligence technology, weight perception based on visual big models has become an important research direction. By combining image processing and natural language processing, visual big models can estimate the weight of objects in images based on input object images. Although visual big models perform well in most visual analysis tasks and exceed human capabilities, the existing visual big models have low accuracy in weight estimation and cannot meet people's needs. How to improve the weight reasoning ability of visual big models is currently a very challenging problem. Summary of the invention

[0003] In order to solve the above technical problems, the present invention proposes a weight estimation method based on a large visual model to solve the problems existing in the above prior art.

[0004] To achieve the above object, the present invention provides a weight estimation method based on a visual large model, comprising:

[0005] Acquire a multimodal weight data set, construct a visual language model, and perform two-stage training on the visual language model based on the multimodal weight data set;

[0006] Obtaining a fine-tuning dataset based on the multimodal weight dataset;

[0007] Obtain a weight estimation model based on the fine-tuning dataset and the visual language model that completed the second stage of training;

[0008] The weight is estimated based on the weight estimation model and the image of the object to be evaluated.

[0009] Optionally, the multimodal weight dataset includes object images, image geology and object weight values.

[0010] An optional, two-stage training process includes:

[0011] The visual language model is trained in the first stage based on the multimodal weight dataset; a multi-expert architecture is introduced into the visual language model after the first stage training, and a second stage training is performed based on the multimodal weight dataset.

[0012] Optionally, the second stage training process includes: introducing a multi-expert architecture into the visual language model trained in the first stage to improve the model, training the improved model based on the multimodal weight dataset, introducing a gating mechanism during training to dynamically select an expert network according to the characteristics of the input data, and updating the gating mechanism parameters, visual language model weights, and expert network weights.

[0013] Optionally, the process of obtaining a fine-tuning dataset includes:

[0014] Based on the category guidance strategy, the categories to which the object images in the multimodal weight dataset belong are annotated; category guidance prompts are generated and added to the multimodal weight dataset, and the processed multimodal weight dataset is converted into a data format.

[0015] Optionally, the process of obtaining a weight estimation model includes:

[0016] Based on the fine-tuning dataset, a reference calibration strategy is adopted to fine-tune the visual language model that has completed the second stage of training to obtain a weight estimation model.

[0017] Optionally, the reference calibration strategy includes two rounds of dialogue. In the first round of dialogue, a reference object image and an object weight value corresponding to the object image are input into the visual language model that has completed the second stage of training for training. In the second round of dialogue, an object image of the same category as the reference object image is input to obtain the object weight.

[0018] The present invention also provides a device, comprising:

[0019] one or more processors;

[0020] Memory; and

[0021] One or more applications, wherein the one or more applications are stored in the memory and are configured to execute the steps of the weight estimation method based on a visual large model by the processor.

[0022] The present invention also provides a storage medium storing a plurality of instructions, wherein the instructions are suitable for loading by a processor to execute the steps in the above-mentioned weight estimation method based on a large visual model.

[0023] Compared with the prior art, the present invention has the following advantages and technical effects:

[0024] The present invention obtains a multimodal weight data set, constructs a visual language model, performs two-stage training on the visual language model based on the multimodal weight data set; obtains a fine-tuning data set based on the multimodal weight data set; obtains a weight estimation model for the visual language model that has completed the second stage of training based on the fine-tuning data set, a category guidance strategy, and a reference calibration strategy; and performs weight estimation based on the weight estimation model and the image of the object to be evaluated. The present invention effectively improves the weight estimation accuracy and weight reasoning ability of the large visual model through the category guidance strategy and the reference calibration strategy. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] The drawings constituting a part of the present application are used to provide a further understanding of the present application. The illustrative embodiments and descriptions of the present application are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0026] Figure 1 The figure is a flow chart of a method according to an embodiment of the present invention. DETAILED DESCRIPTION

[0027] It should be noted that, in the absence of conflict, the embodiments and features in the embodiments of the present application can be combined with each other. The present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0028] It should be noted that the steps shown in the flowcharts of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and that, although a logical order is shown in the flowcharts, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0029] Embodiment 1

[0030] like Figure 1 As shown, this embodiment provides a weight estimation method based on a visual large model, including:

[0031] Acquire a multimodal weight data set, construct a visual language model, and perform two-stage training on the visual language model based on the multimodal weight data set;

[0032] Furthermore, the multimodal weight data set includes object images, image geology and object weight values.

[0033] Furthermore, the two-stage training process includes: performing a first-stage training on the visual language model based on the multimodal weight dataset; introducing a multi-expert architecture into the visual language model after the first-stage training, and performing a second-stage training based on the multimodal weight dataset.

[0034] Furthermore, the second stage training process includes: introducing a multi-expert architecture into the visual language model trained in the first stage to improve the model, training the improved model based on the multimodal weight data set, introducing a gating mechanism during training to dynamically select an expert network according to the characteristics of the input data, and updating the gating mechanism parameters, visual language model weights and expert network weights.

[0035] Specifically, a weight dataset generally includes two parts: object images and object weight values.

[0036] The first stage is the basic pre-training stage, in which the model is pre-trained on a large-scale multimodal dataset (including image and text pairs) to establish basic visual and language understanding capabilities. The second stage introduces the multi-expert (MoE) architecture, where the model dynamically selects suitable expert networks to handle specific tasks through a gating mechanism, thereby improving efficiency and accuracy.

[0037] Obtaining a fine-tuning dataset based on the multimodal weight dataset;

[0038] Furthermore, the process of obtaining a fine-tuning dataset includes:

[0039] Based on the category guidance strategy, the categories to which the object images in the multimodal weight dataset belong are annotated; category guidance prompts are generated and added to the multimodal weight dataset, and the processed multimodal weight dataset is converted into a data format.

[0040] Specifically, the above data cannot be used directly for fine-tuning and training, and data format conversion is required. The multimodal data used for fine-tuning includes prompt information, which is the text or instruction input by the user to the large model to guide the large model to generate the corresponding output. This input can be a paragraph of text, a question, a task description, or even a code snippet. After the image address, weight value, and prompt information are formatted into json format, they are used for model training and fine-tuning.

[0041] Obtain a weight estimation model based on the fine-tuning dataset and the visual language model that completed the second stage of training;

[0042] Furthermore, the process of obtaining the weight estimation model includes: based on the fine-tuning dataset, adopting a reference calibration strategy, fine-tuning the visual language model that has completed the second stage training, and obtaining the weight estimation model.

[0043] In the fine-tuning stage, the processed weight data is used to enhance the performance of the model on the weight estimation task. Further training and optimization of the model usually uses specific task datasets to enhance its performance on specific tasks.

[0044] Furthermore, the reference calibration strategy includes two rounds of dialogue. In the first round of dialogue, a reference object image and an object weight value corresponding to the object image are input into the visual language model that has completed the second stage of training for training. In the second round of dialogue, an object image of the same category as the reference object image is input to obtain the object weight.

[0045] Specifically, the design of the prompt plays a great role in whether the visual model can accurately estimate the weight of the object. The common prompt only asks the visual model whether the objects in the image are different. This embodiment designs a new strategy - category guidance strategy, which is used to guide the visual model to better estimate the weight of the object.

[0046] Specifically, instead of asking the big visual model to estimate the weight of an object directly, we modify the prompts to first guide the big visual model to determine the category of the object in the image, and then estimate the weight of the object based on this. By guiding the big visual model to consider the category of the object first, we can help it limit the uncertainty of the estimate to a reasonable range based on the prior category information, thereby reducing the estimation error.

[0047] A category-guided strategy is used to generate multimodal data for model training and fine-tuning. The core data input to the large visual model consists of two parts: one is the picture, and the other is the prompt information. The picture is obtained from the weighted dataset, and the prompt is a specifically designed category-guided prompt.

[0048] The second strategy designed in this embodiment is the reference calibration strategy. Specifically, before asking the visual big model to estimate the weight of the target image, first provide a reference image of the same category and its true weight value. In the form of two rounds of dialogue, the first round of dialogue inputs a reference image and the true weight value of the object in the image to the visual big model, and the second round of dialogue allows the visual big model to predict the weight of the same category of objects as the reference image based on previous information. In this way, the visual big model can refer to an example to calibrate its estimate of the test image, thereby improving accuracy.

[0049] The weight is estimated based on the weight estimation model and the image of the object to be evaluated.

[0050] The experimental results show that the weight estimation accuracy of the visual large model is greatly improved by using the category guidance strategy to train the visual large model, compared with not using this strategy for training. The improvement is greater than 5%. Similarly, the weight estimation accuracy of the visual large model is also greatly improved by using the reference calibration strategy to train the visual large model, compared with not using this strategy for training. The improvement is greater than 5%.

[0051] In this embodiment, a device is provided, which includes: one or more processors; a memory; and one or more applications, wherein the one or more applications are stored in the memory and are configured so that the processor executes the steps in the above-mentioned weight estimation method based on the visual large model.

[0052] In this embodiment, a storage medium is provided, which stores a plurality of instructions, and the instructions are suitable for a processor to load to execute the steps in the above-mentioned weight estimation method based on a large visual model.

[0053] The above are only preferred specific implementations of the present application, but the protection scope of the present application is not limited thereto. Any changes or substitutions that can be easily thought of by a person skilled in the art within the technical scope disclosed in the present application should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A weight estimation method based on a visual large model, characterized in that: The following steps are involved: Acquire a multimodal weight data set, construct a visual language model, and perform two-stage training on the visual language model based on the multimodal weight data set; Obtaining a fine-tuning dataset based on the multimodal weight dataset; Obtain a weight estimation model based on the fine-tuning dataset and the visual language model that completed the second stage of training; The weight is estimated based on the weight estimation model and the image of the object to be evaluated.

2. The weight estimation method based on visual large model according to claim 1 is characterized in that: The multimodal weight dataset includes object images, image geology, and object weight values.

3. The weight estimation method based on visual large model according to claim 1 is characterized in that: The two-stage training process includes: The visual language model is trained in the first stage based on the multimodal weight dataset; a multi-expert architecture is introduced into the visual language model after the first stage training, and a second stage training is performed based on the multimodal weight dataset.

4. The weight estimation method based on visual large model according to claim 3 is characterized in that: The second stage training process includes: introducing a multi-expert architecture into the visual language model trained in the first stage to improve the model, training the improved model based on the multimodal weight dataset, introducing a gating mechanism during training to dynamically select an expert network according to the characteristics of the input data, and updating the gating mechanism parameters, visual language model weights and expert network weights.

5. The weight estimation method based on visual large model according to claim 2 is characterized in that: The process of obtaining a fine-tuning dataset includes: Based on the category guidance strategy, the categories to which the object images in the multimodal weight dataset belong are annotated; category guidance prompts are generated and added to the multimodal weight dataset, and the processed multimodal weight dataset is converted into a data format.

6. The weight estimation method based on visual large model according to claim 1 is characterized in that: The process of obtaining a weight estimation model includes: Based on the fine-tuning dataset, a reference calibration strategy is adopted to fine-tune the visual language model that has completed the second stage of training to obtain a weight estimation model.

7. The weight estimation method based on visual large model according to claim 6 is characterized in that: The reference calibration strategy includes two rounds of dialogue. In the first round of dialogue, a reference object image and the object weight value corresponding to the object image are input into the visual language model that has completed the second stage of training for training. In the second round of dialogue, an object image of the same category as the reference object image is input to obtain the object weight.

8. A device, characterized in that: The device comprises: one or more processors; Memory; and One or more applications, wherein the one or more applications are stored in the memory and are configured to execute, by the processor, the steps of the weight estimation method based on a visual large model as claimed in any one of claims 1 to 7.

9. A storage medium, characterized in that: The storage medium stores a plurality of instructions, and the instructions are suitable for being loaded by a processor to execute the steps in the weight estimation method based on a visual large model according to any one of claims 1 to 7.

Images