Fine adjustment and linear detection method and device for multi-modal large model and medium
By adding linear target marks to the vocabulary list of multimodal large models and performing fine-tuning training, the problem of low accuracy in online shape detection of multimodal large models is solved, and the accurate identification of linear coordinates is achieved.
Patent Information
- Application Number
- CN202411082753.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-07
- Publication Date
- 2024-12-10
AI Technical Summary
The existing multimodal large model can only determine the rectangular box containing the line shape during the detection of the line shape, resulting in low accuracy of the recognition result and cannot meet the scenarios where the accuracy of the line position information is high.
By receiving the target identifier for representing the starting and ending points of the line shape, adding the target identifier to the vocabulary list of the multimodal large model, and fine-tuning training is performed using the pre-saved data set, so that the multimodal large model can output the coordinate information of the line shape.
The accuracy and accuracy in multimodal large model line shape recognition is improved, so that it can accurately determine the starting and ending coordinates of the line shape.
Smart Images
Figure CN119107469A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of image processing technology, and in particular to a method, device and medium for fine-tuning and linear detection of a multi-modal large model. Background Art
[0002] Linear detection has a wide range of applications in many fields, among which it has important application value in the fields of autonomous driving, intelligent traffic monitoring and driver assistance systems. However, when performing line detection on an image, the existing multimodal large model can only determine the detection frame containing the line and use the detection frame as the position information corresponding to the line. The position information identified by this line detection method is not accurate and cannot be applied to scenes that require high accuracy of the line position information.
[0003] Therefore, how to improve the accuracy of large multimodal models in linear detection has become an urgent problem to be solved. Summary of the invention
[0004] The present application provides a method, device and medium for fine-tuning and linear detection of a large multimodal model, so as to solve the problem in the prior art that a large multimodal model performs linear recognition in the form of a rectangular frame, resulting in low accuracy of recognition results.
[0005] In a first aspect, an embodiment of the present application provides a method for fine-tuning a multimodal large model, the method comprising:
[0006] Receive input target identifiers for indicating the start and end points of a line, and add the target identifiers to a vocabulary corresponding to the multimodal large model;
[0007] Acquire a pre-saved data set, wherein the data set includes a sample image annotated with linear sample position information, wherein the linear sample position information includes the target identifier and sample coordinates of two endpoints of the linear;
[0008] The data set is used to perform fine-tuning training on the multimodal large model, so that the multimodal large model can perform linear recognition based on the target identifier.
[0009] In a second aspect, an embodiment of the present application further provides a line shape detection method based on a multimodal large model, wherein the multimodal large model is trained based on any of the above methods, and the method includes:
[0010] Inputting a target image to be subjected to line shape detection into a multimodal large model, so that the multimodal large model performs line shape recognition on the target image;
[0011] The position information corresponding to each line shape in the target image output by the multimodal large model is obtained, wherein the position information corresponding to each line shape includes the target identifier for indicating the starting and ending points of the line shape and the coordinate information of the two endpoints of the line shape.
[0012] In a third aspect, an embodiment of the present application further provides an electronic device, comprising at least a processor and a memory, wherein the processor is used to implement the steps of any of the above-mentioned methods for fine-tuning a multimodal large model or any of the above-mentioned methods for linear detection based on a multimodal large model when executing a computer program stored in the memory.
[0013] In a fourth aspect, an embodiment of the present application further provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps of any of the above-mentioned methods for fine-tuning a multimodal large model or any of the above-mentioned methods for linear detection based on a multimodal large model.
[0014] In an embodiment of the present application, an electronic device receives an input target identifier for indicating the start and end points of a line, adds the target identifier to a vocabulary corresponding to a multimodal large model, obtains a pre-saved data set, wherein the data set includes a sample image with sample position information of a line, and the sample position information of the line includes the target identifier and sample coordinates of the two end points of the line; and uses the data set to fine-tune the multimodal large model so that the multimodal large model can perform line recognition based on the target identifier. In an embodiment of the present application, the multimodal large model is fine-tuned by using a pre-configured target identifier for indicating the start and end points of a line, and by using a sample image with sample position information of a line, so that the multimodal large model has the ability to output line coordinates, thereby improving the accuracy and precision of the multimodal large model's line recognition. BRIEF DESCRIPTION OF THE DRAWINGS
[0015] In order to more clearly illustrate the technical solution of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.
[0016] Figure 1 A schematic diagram of a fine-tuning process of a multi-modal large model provided in an embodiment of the present application;
[0017] Figure 2 A schematic diagram of a linear detection process based on a multimodal large model provided in an embodiment of the present application;
[0018] Figure 3 A schematic diagram of a target image provided in an embodiment of the present application;
[0019] Figure 4 A schematic diagram of the structure of a fine-tuning device for a multi-modal large model provided in an embodiment of the present application;
[0020] Figure 5 A schematic diagram of the structure of a linear detection device based on a multi-modal large model provided in an embodiment of the present application;
[0021] Figure 6 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION
[0022] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application will be further described in detail below in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present application.
[0023] In order to enable the multimodal large model to have linear recognition capabilities and improve the linear recognition accuracy of the multimodal large model, the embodiments of the present application provide a multimodal large model fine-tuning, linear detection method, device and medium.
[0024] In an embodiment of the present application, an input target identifier for representing the starting and ending points of a line is received, and the target identifier is added to a vocabulary corresponding to the multimodal large model; a pre-saved data set is obtained, wherein the data set includes sample images marked with sample position information of the line, and the sample position information of the line includes the target identifier and sample coordinates of the two endpoints of the line; and the data set is used to fine-tune the multimodal large model so that the multimodal large model recognizes the line based on the target identifier.
[0025] Figure 1 A schematic diagram of a fine-tuning process of a multimodal large model provided in an embodiment of the present application, the process comprising:
[0026] S101: receiving an input target identifier for indicating the start and end points of a line, and adding the target identifier to a vocabulary corresponding to a multimodal large model.
[0027] A method for fine-tuning a multimodal large model provided in an embodiment of the present application is applied to an electronic device, which may be a PC or a server.
[0028] The existing multimodal large model only has the ability to detect rectangular boxes, that is, to generate rectangular box coordinates of the detection target, but the existing multimodal large model cannot generate linear coordinates.
[0029] Based on this, in order to enable the multimodal large model to have the ability to detect lines and to determine and output the coordinates of the lines, in an embodiment of the present application, the electronic device receives an input target identifier for representing the starting and ending points of the line and adds the target identifier to the vocabulary corresponding to the multimodal large model.
[0030] Among them, the vocabulary corresponding to the multimodal large model contains all the characters that the multimodal large model can determine and output, and the target identifier representing the starting and ending points of the line is added to the vocabulary, so that the multimodal large model can determine the starting and ending points of the line, and output the coordinates of the starting and ending points of the line through the target identifier, thereby realizing the ability of the output multimodal large model to determine and output the coordinates of the line.
[0031] Taking the multimodal large model Emu2 as an example, the original vocabulary of Emu2 contains 32274 characters. Based on this, on the basis of the disclosure of the embodiments of the present application, the target identification of the linear start and end points can be added to the vocabulary positions 32274 and 32275. After the target identification is added, the vocabulary of Emu2 is increased to 32276 characters.
[0032] It should be noted that, because the target identifier includes a first sub-identifier for indicating the starting point of the line and a second sub-identifier for indicating the end point of the line, adding the target identifier to the vocabulary needs to occupy two vocabulary positions.
[0033] S102: Acquire a pre-saved data set, wherein the data set includes a sample image annotated with linear sample position information, and the linear sample position information includes the target identifier and sample coordinates of two endpoints of the line.
[0034] In an embodiment of the present application, a data set is stored in the electronic device, and the data set is used to fine-tune the multimodal large model.
[0035] The data set includes multiple sample images, and each sample image is annotated with linear sample position information. The linear sample position information includes target identifiers for indicating the start and end points of the line and sample coordinates of the two end points of the line.
[0036] For example, the sample location information can be <line> (x1,y1),(x2,y2)< / line> ,in, <line> and< / line> is the target identifier representing the starting and ending points of the line, and (x1, y1) and (x2, y2) are the sample coordinates of the two endpoints of the line.
[0037] S103: fine-tune the multimodal large model using the data set, so that the multimodal large model can perform line recognition based on the target identifier.
[0038] In an embodiment of the present application, the electronic device uses a data set to fine-tune and train the multimodal large model, so that the multimodal large model can recognize the line shape according to the target identifiers used to represent the starting and ending points of the line shape.
[0039] Specifically, the electronic device inputs each sample image in the data set and a prompt word prompting the multimodal large model to perform line recognition into the multimodal large model, and the multimodal large model performs line recognition on the sample image according to the prompt word. The electronic device determines a loss value according to the line recognition result of each sample image by the multimodal large model and the sample position information marked in each sample image, and fine-tunes the multimodal large model based on the loss value.
[0040] Among them, when the electronic device fine-tunes the multimodal large model, it can use the LoRA fine-tuning method or other fine-tuning methods, which are not limited here.
[0041] In an embodiment of the present application, the multimodal large model is fine-tuned by pre-configured target identifiers for indicating the starting and ending points of a line and by using sample images marked with sample position information of the line, so that the multimodal large model has the ability to output linear coordinates, thereby improving the accuracy and precision of the multimodal large model's linear recognition.
[0042] In order to enable the multimodal large model to have linear recognition capability and improve the linear recognition accuracy of the multimodal large model, based on the above embodiment, in the embodiment of the present application, before fine-tuning the multimodal large model using the data set, the method further includes:
[0043] According to the interval range corresponding to the number of annotated lines in each sample image in the data set, the sample images belonging to the same interval range are split into corresponding sub-data sets;
[0044] Sort each sub-data set according to the maximum value of the interval range corresponding to each sub-data set;
[0045] According to the sorting result, for each sub-dataset in turn, the sub-dataset and other sub-datasets sorted before the sub-dataset in the sorting result are used as data sets, and a subsequent process of fine-tuning the multimodal large model using the data sets is performed.
[0046] Since the multimodal large model does not have the ability to detect linear shapes, directly training sample images containing multiple linear annotations will not produce good training results. Therefore, in order to enable the multimodal large model to quickly learn the linear features, in an embodiment of the present application, the multimodal large model can be fine-tuned in a staged manner. As the number of stages increases, the number of linear shapes contained in the sample images used to fine-tune the multimodal large model in each stage gradually increases.
[0047] Specifically, in the embodiment of the present application, the electronic device pre-stores each interval range, and the electronic device splits the data set according to each interval range and the number of line shapes contained in each sample image in the data set to obtain multiple sub-data sets. The electronic device sorts each sub-data set and fine-tunes the multimodal large model according to the sorting result.
[0048] Specifically, the electronic device splits the sample images belonging to the same interval range into corresponding sub-datasets according to the interval range corresponding to the number of labeled lines in each sample image in the data set; the electronic device sorts each sub-dataset according to the maximum value of the interval range corresponding to each sub-dataset; according to the sorting result, for each sub-dataset in turn, the sub-dataset and other sub-datasets sorted before the sub-dataset in the sorting result are used as data sets, and the multimodal large model is fine-tuned and trained using the data sets.
[0049] For example, the electronic device divides the fine-tuning process of the multimodal large model into three stages (easy, medium, and difficult) according to the difficulty level. The data set used in the first stage is the sub-dataset p1, and the interval range corresponding to the sub-dataset p1 is 1-4, that is, the sub-dataset p1 contains sample images with corresponding number of line shapes of 1-4; the data set of the second stage is the sub-dataset p1+sub-dataset p2, and the interval range corresponding to the sub-dataset p2 is 5-8, that is, the sub-dataset p2 contains sample images with corresponding number of line shapes of 5-8; the data set of the third stage is the sub-dataset p1+sub-dataset p2+sub-dataset p3, which contains all sample images.
[0050] By fine-tuning the multimodal large model through the above method, the learning of the multimodal large model can be made from easy to difficult, making it easier for the multimodal large model to learn the characteristics of linear targets, which can effectively avoid the risk of loss not decreasing.
[0051] In order to enable the multimodal large model to have linear recognition capability and improve the linear recognition accuracy of the multimodal large model, on the basis of the above embodiments, in the embodiment of the present application, the use of the data set to perform fine-tuning training on the multimodal large model includes:
[0052] For each sample image contained in the data set, the sample image is input into the multimodal large model to obtain a first prediction result output by the multimodal large model; a first loss value is determined based on the first prediction result and the linear sample position information corresponding to the sample image; and the parameters of the multimodal large model are adjusted based on the first loss value.
[0053] In an embodiment of the present application, when the electronic device fine-tunes the multimodal large model, the electronic device inputs each sample image contained in the data set into the multimodal large model to obtain a first prediction result output by the multimodal large model; determines a first loss value based on the first prediction result and the linear sample position information corresponding to the sample image; and adjusts the parameters of the multimodal large model based on the first loss value.
[0054] In order to enable the multimodal large model to have linear recognition capability and improve the linear recognition accuracy of the multimodal large model, on the basis of the above embodiments, in the embodiment of the present application, the first prediction result is the confidence corresponding to each character in the vocabulary; wherein the character is a coordinate or an identifier used to represent any recognition result;
[0055] Determining the first loss value according to the first prediction result and the linear sample position information corresponding to the sample image includes:
[0056] Obtaining a first confidence level of the target identifier in the first prediction result, and a second confidence level and a third confidence level of the sample coordinates of the two endpoints;
[0057] The first loss value is determined according to the first confidence level, the second confidence level and the third confidence level.
[0058] In an embodiment of the present application, when the multimodal large model is performing fine-tuning training, the first prediction result output by the multimodal large model is the output confidence corresponding to each character in the vocabulary corresponding to the multimodal large model, wherein each character in the vocabulary is a coordinate or an identifier used to represent any recognition result, such as a target identifier corresponding to a line, an identifier corresponding to a rectangular box, etc.
[0059] In addition, in the embodiment of the present application, when a character is a coordinate, the character is a specific numerical value and an identifier of the horizontal coordinate or an identifier of the vertical coordinate.
[0060] In an embodiment of the present application, when the electronic device determines the first loss value based on the first prediction result and the linear sample position information corresponding to the sample image, the electronic device obtains the first confidence of the target identification in the first prediction result, the second confidence and the third confidence of the sample coordinates of the two endpoints; and determines the first loss value based on the first confidence, the second confidence and the third confidence.
[0061] Among them, the confidences corresponding to the target identifier and the sample coordinates of the two endpoints in the sample position information of the sample image are both maximum confidence values.
[0062] Based on this, when the electronic device determines the first loss value based on the first confidence, the second confidence and the third confidence, the electronic device can determine the difference between the maximum confidence value and the first confidence, the second confidence and the third confidence, and determine the sum of the differences as the first loss value.
[0063] In order to enable the multimodal large model to have linear recognition capabilities and improve the linear recognition accuracy of the multimodal large model, on the basis of the above embodiments, in the embodiment of the present application, if the data set also includes other samples and corresponding sample annotations for training the multimodal large model to learn other capabilities, the fine-tuning training of the multimodal large model using the data set includes:
[0064] Obtaining a saved first weight and a second weight, wherein the first weight is greater than the second weight;
[0065] Inputting the sample image into the multimodal large model, obtaining a second prediction result output by the multimodal large model, and determining a second loss value according to the second prediction result, the linear sample position information corresponding to the sample image, and the first weight;
[0066] Inputting the other samples into the multimodal large model, obtaining a third prediction result output by the multimodal large model, and determining a third loss value according to the third prediction result, the sample labels corresponding to the other samples, and the second weight;
[0067] According to the second loss value and the third loss value, the parameters of the multimodal large model are adjusted.
[0068] Since the multimodal large model can be applied to various fields, the data set corresponding to the multimodal large model may not only contain sample images for fine-tuning the linear recognition ability of the multimodal large model, but also other samples and corresponding sample annotations for training the multimodal large model to learn other capabilities.
[0069] In order to increase the attention of the multimodal large model to the linear recognition capability and obtain better linear recognition effect, in an embodiment of the present application, the weight corresponding to each capability can be pre-configured when calculating the loss value, and the first weight corresponding to the linear recognition capability can be set greater than the second weight corresponding to other capabilities.
[0070] Specifically, in an embodiment of the present application, if the electronic device determines that the data set also includes other samples and corresponding sample annotations for training the multimodal large model to learn other capabilities, the electronic device obtains the saved first weight and second weight.
[0071] The electronic device inputs a sample image into the multimodal large model, obtains a second prediction result output by the multimodal large model, and determines a second loss value based on the second prediction result, linear sample position information corresponding to the sample image, and the first weight; the electronic device inputs other samples into the multimodal large model, obtains a third prediction result output by the multimodal large model, and determines a third loss value based on the third prediction result, sample labels corresponding to other samples, and the second weight; and adjusts the parameters of the multimodal large model based on the second loss value and the third loss value.
[0072] Specifically, in an embodiment of the present application, when the electronic device determines the second loss value, the electronic device obtains the fourth confidence of the target identifier in the second prediction result, the fifth confidence and the sixth confidence of the sample coordinates of the two endpoints; determines each difference between the maximum confidence value and the fourth confidence, the fifth confidence and the sixth confidence, and determines each first product of the difference and the first weight, and determines the sum of each first product as the second loss value.
[0073] Furthermore, in the embodiment of the present application, the first weight may also include sub-weights corresponding to the target identifier and the endpoints of the two samples, respectively, and the corresponding sub-weights are multiplied by the corresponding confidence levels.
[0074] When the electronic device determines the third loss value, the electronic device obtains the seventh confidence level outputted by the sample annotations corresponding to other samples in the third prediction result, determines the difference between the maximum confidence level and the seventh confidence level, and determines the second product of the difference and the second weight as the third loss value. If the sample annotation includes multiple characters in the vocabulary, the process of determining the third loss value is the same as the process of determining the second loss value, and will not be described in detail here.
[0075] In addition, in the embodiment of the present application, when the electronic device saves the first weight and the second weight, the first weight and the second weight can be saved in a weight vector, each dimension of the weight vector corresponds to a character in the vocabulary, and the electronic device can obtain the first weight corresponding to each sample according to the annotated content corresponding to the sample. Generally, in the embodiment of the present application, the value of the dimension corresponding to the character related to line recognition in the weight vector is set to 2, and the values of other dimensions are set to 1.
[0076] In order to enable the multimodal large model to have linear recognition capabilities and improve the linear recognition accuracy of the multimodal large model, on the basis of the above embodiments, in an embodiment of the present application, the target identifier includes a first sub-identifier for indicating the starting point of the line and a second sub-identifier for indicating the end point of the line.
[0077] In the embodiment of the present application, the target identifier includes a first sub-identifier for indicating a linear starting point and a second sub-identifier for indicating an linear end point.
[0078] For example, the target identifier is <line>< / line> ,in <line> is the first sub-identifier indicating the starting point of the line.< / line> The second sub-identifier indicating the end point of the line, that is, <line> and< / line> The coordinate information of the two points is in a linear form, and the electronic device will add two new characters to the vocabulary of the multimodal large model.
[0079] On the basis of the above embodiments, the present application also provides a line shape detection method based on a multi-modal large model. Figure 2 A schematic diagram of a linear detection process based on a multimodal large model provided in an embodiment of the present application, the process includes:
[0080] S201: Inputting a target image to be subjected to line shape detection into a multimodal large model, so that the multimodal large model performs line shape recognition on the target image.
[0081] A line shape detection method based on a multimodal large model provided in an embodiment of the present application is applied to an electronic device, which may be a PC or a server.
[0082] In an embodiment of the present application, the multi-modal large model is based on the fine-tuning method of the above-mentioned embodiments. After the fine-tuning is completed, the multi-modal large model can be used to perform line detection on the target image.
[0083] Specifically, in an embodiment of the present application, the electronic device inputs a target image to be subjected to line shape detection into a multimodal large model, so that the multimodal large model performs line shape recognition on the target image.
[0084] S202: Obtaining position information corresponding to each line shape in the target image output by the multimodal large model, wherein the position information corresponding to each line shape includes the target identifier for indicating the starting and ending points of the line shape and the coordinate information of the two endpoints of the line shape.
[0085] In the embodiment of the present application, since the vocabulary corresponding to the multimodal large model contains target identifiers for indicating the start and end points of a line shape, and the multimodal large model has learned the ability to detect line shapes, based on this, the multimodal large model can determine and output the position information corresponding to each line shape in the target image according to the target identifier.
[0086] In the embodiment of the present application, the position information corresponding to each line includes target identifiers for indicating the starting and ending points of the line and coordinate information of the two endpoints of the line.
[0087] In addition, in the embodiment of the present application, each coordinate information can be split into a horizontal coordinate and a vertical coordinate, and the horizontal coordinate is sorted first by default.
[0088] For example, the position information output by the multimodal large model is <line><patch_index_0209><patch_index_0111><patch_index_0223><patch_index_0128>< / line> ,in, <line>< / line> For target identification, <line> is the first sub-identifier indicating the starting point of the line.< / line> The second sub-identifier indicating the end point of the line;<patch_index_0209><patch_index_0111> is the coordinate information of an endpoint,<patch_index_0209> is the horizontal coordinate of the coordinate information,<patch_index_0111> is the ordinate of the coordinate information;<patch_index_0223><patch_index_0128> is the coordinate information of the other endpoint,<patch_index_0223> is the horizontal coordinate of the coordinate information,<patch_index_0128> It is the vertical coordinate of the coordinate information.
[0089] In order to enable the multimodal large model to have linear recognition capability and improve the linear recognition accuracy of the multimodal large model, on the basis of the above embodiments, in the embodiment of the present application, the method further includes:
[0090] According to the position information corresponding to each line shape in the target image, a description text corresponding to the target image is generated.
[0091] In the embodiment of the present application, the electronic device may also generate a description text corresponding to the target image according to the position information corresponding to each line shape in the target image.
[0092] Specifically, the electronic device may input the position information into the language macro model, and control the language macro model through the prompt words to generate and output a description text of the target image to which the position information belongs according to the position information.
[0093] In addition, the electronic device can also directly input the target image into the multimodal large model, and through prompt words, enable the multimodal large model to determine the position information corresponding to each line in the target image, and generate a description text corresponding to the target image based on the position information.
[0094] In the embodiment of the present application, the following example is taken as an example of a multimodal large model in the field of transportation:
[0095] The perception capabilities of the multimodal large model in the transportation field include the perception of people, motor vehicles, non-motor vehicles, and traffic elements in traffic scenes, among which traffic elements include traffic signs, traffic facilities (crash barrels, water barriers, etc.), traffic equipment (signal lights, etc.), road signs (guide arrows, speed limit signs, etc.), and road markings (lane lines, etc.). Existing multimodal models in the transportation field do not support the output of linear coordinates. If a rectangular frame is used to represent lane lines, it will lead to problems such as the target frame being too large, the target frame features being unclear, and the model being difficult to learn. Therefore, by constructing a lane line dataset, increasing linear output, and giving the multimodal large model in the transportation field the ability to output linear coordinates, it can help the multimodal large model in the transportation field better learn the characteristics of lane lines, and at the same time deepen the perception capabilities of the multimodal large model in the transportation field.
[0096] Figure 3 A schematic diagram of a target image provided in an embodiment of the present application, wherein the target image contains five lane lines. If the target image is input into a fine-tuned multimodal large model for line shape detection, the output of the multimodal large model is: <line><patch_index_0022><patch_index_0222><patch_index_0066><patch_index_0111>< / line> <line><patch_index_0081><patch_index_0222><patch_index_0096><patch_index_0111>< / line> <line><patch_index_0139><patch_index_0223><patc h_index_0126><patch_index_0112>< / line> <line><patch_index_0154><patch_inde x_0111><patch_index_0195><patch_index_0223>< / line> <line><patch_index_0180><patch_index_0111><patch_index_0222><patch_index_0181>< / line> <line><pa tch_index_0209><patch_index_0111><patch_index_0223><patch_index_0128>< / line> <line><patch_index_0000><patch_index_0165><patch_index_0023><patch_index_0130>< / line> .
[0097] In addition, in the embodiment of the present application, the multimodal large model can also input linear types, such as <ref> straight line< / ref> <line> (x1,y1),(x2,y2)< / line> , <ref> curve< / ref> <line> (x1,y1),(x2,y2),(x3,y3),(x4,y4)< / line> .
[0098] Based on the above embodiments, Figure 4 A schematic diagram of the structure of a fine-tuning device for a multi-modal large model provided in an embodiment of the present application, the device comprising:
[0099] The processing module 401 is used to receive an input target identifier for indicating the start and end points of a line, and add the target identifier to a vocabulary corresponding to the multimodal large model; obtain a pre-saved data set, wherein the data set includes a sample image annotated with sample position information of the line, and the sample position information of the line includes the target identifier and sample coordinates of two end points of the line;
[0100] The fine-tuning module 402 is used to use the data set to perform fine-tuning training on the multimodal large model, so that the multimodal large model can perform line recognition based on the target identifier.
[0101] In a possible implementation, the processing module 401 is also used to split the sample images belonging to the same interval range into corresponding sub-datasets according to the interval range corresponding to the number of labeled lines in each sample image in the data set; sort each sub-dataset according to the maximum value of the interval range corresponding to each sub-dataset and execute the subsequent process of fine-tuning the multimodal large model using the data set.
[0102] In a possible implementation, the processing module 401 is also used to input each sample image included in the data set into the multimodal large model to obtain a first prediction result output by the multimodal large model; determine a first loss value based on the first prediction result and the linear sample position information corresponding to the sample image; and adjust the parameters of the multimodal large model based on the first loss value.
[0103] In a possible implementation, the first prediction result is the confidence level corresponding to each character in the vocabulary; wherein the character is a coordinate or an identifier used to represent any recognition result;
[0104] The fine-tuning module 402 is specifically used to obtain the first confidence of the target identifier in the first prediction result, the second confidence and the third confidence of the sample coordinates of the two endpoints; and determine the first loss value based on the first confidence, the second confidence and the third confidence.
[0105] In a possible implementation, the fine-tuning module 402 is also used to obtain the saved first weight and second weight if the data set also includes other samples and corresponding sample annotations for training the multimodal large model to learn other capabilities, wherein the first weight is greater than the second weight; input the sample image into the multimodal large model to obtain the second prediction result output by the multimodal large model, and determine the second loss value based on the second prediction result, the linear sample position information corresponding to the sample image, and the first weight; input the other samples into the multimodal large model to obtain the third prediction result output by the multimodal large model, and determine the third loss value based on the third prediction result, the sample annotations corresponding to the other samples, and the second weight; adjust the parameters of the multimodal large model based on the second loss value and the third loss value.
[0106] In a possible implementation manner, the target identifier includes a first sub-identifier for indicating a linear starting point and a second sub-identifier for indicating an linear end point.
[0107] Based on the above embodiments, Figure 5A schematic diagram of the structure of a linear detection device based on a multimodal large model provided in an embodiment of the present application, the device comprising:
[0108] A processing module 502 is used to input a target image to be subjected to line shape detection into a multimodal large model, so that the multimodal large model performs line shape recognition on the target image;
[0109] The detection module 503 is used to obtain the position information corresponding to each line shape in the target image output by the multimodal large model, wherein the position information corresponding to each line shape includes the target identifier used to represent the starting and ending points of the line shape and the coordinate information of the two endpoints of the line shape.
[0110] In a possible implementation, the processing module 502 is further configured to generate a description text corresponding to the target image according to the position information corresponding to each line shape in the target image.
[0111] Based on the above embodiments, the present application also provides an electronic device, Figure 6 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application is shown in FIG. Figure 6 As shown, it includes: a processor 601, a communication interface 602, a memory 603 and a communication bus 604, wherein the processor 601, the communication interface 602, and the memory 603 communicate with each other through the communication bus 604;
[0112] The memory 603 stores a computer program. When the program is executed by the processor 601, the processor 601 executes the steps of the fine-tuning method of the multimodal large model or the linear detection method based on the multimodal large model as provided in the above embodiments.
[0113] Since the principle of solving the problem by the above electronic device is similar to the fine-tuning method of the multimodal large model or the linear detection method based on the multimodal large model, the implementation of the above electronic device can refer to the embodiment of the method, and the repeated parts will not be repeated.
[0114] The communication bus mentioned in the above electronic device can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The communication bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus. The communication interface 602 is used for communication between the above electronic device and other devices. The memory may include a random access memory (RAM) and may also include a non-volatile memory (NVM), such as at least one disk storage. Optionally, the memory may also be at least one storage device located away from the aforementioned processor.
[0115] The above-mentioned processor can be a general-purpose processor, including a central processing unit, a network processor (Network Processor, NP), etc.; it can also be a digital signal processing processor (Digital Signal Processing, DSP), an application-specific integrated circuit, a field programmable gate array or other programmable logic device, a discrete gate or transistor logic device, a discrete hardware component, etc.
[0116] On the basis of the above embodiments, an embodiment of the present invention further provides a computer-readable storage medium, which stores a computer program executable by a processor. When the program runs on the processor, the processor implements the fine-tuning method of the multimodal large model or the linear detection method steps based on the multimodal large model as provided in the above embodiments.
[0117] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.
[0118] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.
[0119] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.
[0120] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.
[0121] Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is also intended to include these modifications and variations.
Claims
1. A fine-tuning method for a multimodal large model, characterized in that: The method comprises: Receive input target identifiers for indicating the start and end points of a line, and add the target identifiers to a vocabulary corresponding to the multimodal large model; Acquire a pre-saved data set, wherein the data set includes a sample image annotated with linear sample position information, wherein the linear sample position information includes the target identifier and sample coordinates of two endpoints of the linear; The data set is used to perform fine-tuning training on the multimodal large model, so that the multimodal large model can perform linear recognition based on the target identifier.
2. The method according to claim 1, characterized in that Before fine-tuning the multimodal large model using the data set, the method further includes: According to the interval range corresponding to the number of annotated lines in each sample image in the data set, the sample images belonging to the same interval range are split into corresponding sub-data sets; According to the maximum value of the interval range corresponding to each sub-dataset, each sub-dataset is sorted and a subsequent process of using the data set to fine-tune the multimodal large model is performed.
3. The method according to claim 1 or 2, characterized in that: The fine-tuning training of the multimodal large model using the data set includes: For each sample image contained in the data set, the sample image is input into the multimodal large model to obtain a first prediction result output by the multimodal large model; a first loss value is determined based on the first prediction result and the linear sample position information corresponding to the sample image; and the parameters of the multimodal large model are adjusted based on the first loss value.
4. The method according to claim 3, characterized in that The first prediction result is the confidence level corresponding to each character in the vocabulary; wherein the character is a coordinate or an identifier used to represent any recognition result; Determining the first loss value according to the first prediction result and the linear sample position information corresponding to the sample image includes: Obtaining a first confidence level of the target identifier in the first prediction result, and a second confidence level and a third confidence level of the sample coordinates of the two endpoints; The first loss value is determined according to the first confidence level, the second confidence level and the third confidence level.
5. The method according to claim 1, characterized in that If the data set also includes other samples and corresponding sample annotations for training the multimodal large model to learn other capabilities, the fine-tuning training of the multimodal large model using the data set includes: Obtaining a saved first weight and a second weight, wherein the first weight is greater than the second weight; Inputting the sample image into the multimodal large model, obtaining a second prediction result output by the multimodal large model, and determining a second loss value according to the second prediction result, the linear sample position information corresponding to the sample image, and the first weight; Inputting the other samples into the multimodal large model, obtaining a third prediction result output by the multimodal large model, and determining a third loss value according to the third prediction result, the sample labels corresponding to the other samples, and the second weight; According to the second loss value and the third loss value, the parameters of the multimodal large model are adjusted.
6. The method according to claim 1, characterized in that The target identifier includes a first sub-identifier for indicating a linear start point and a second sub-identifier for indicating a linear end point.
7. A linear detection method based on a multimodal large model, characterized in that: The multimodal large model is trained based on the method according to any one of claims 1 to 6, and the method comprises: Inputting a target image to be subjected to line shape detection into a multimodal large model, so that the multimodal large model performs line shape recognition on the target image; The position information corresponding to each line shape in the target image output by the multimodal large model is obtained, wherein the position information corresponding to each line shape includes the target identifier for indicating the starting and ending points of the line shape and the coordinate information of the two endpoints of the line shape.
8. The method according to claim 7, characterized in that The method further comprises: According to the position information corresponding to each line shape in the target image, a description text corresponding to the target image is generated.
9. An electronic device, characterized in that: The electronic device includes at least a processor and a memory, and the processor is used to implement the fine-tuning method of the multimodal large model as described in any one of claims 1-6, or the steps of the line detection method based on the multimodal large model as described in any one of claims 7-8 when executing the computer program stored in the memory.
10. A computer-readable storage medium, characterized in that: It stores a computer program, which, when executed by a processor, implements the fine-tuning method of the multimodal large model as described in any one of claims 1-6, or the steps of the line detection method based on the multimodal large model as described in any one of claims 7-8.