Open question answering and training method and device of multi-modal large model and related equipment

Abstract

The application discloses a kind of open question and the training method, device and related equipment of multimodal large model, to promote multimodal large model to focus on spatial information, matched image description text with spatial information is generated for training image in pre-training phase, spatial information is used to indicate the spatial position of object contained in training image in training image, and the pre-training of multimodal large model is carried out using training image and the above-mentioned image description text with explicit object spatial information, can make that multimodal large model is further focused on the spatial position of object in image on the basis of learning the semantic alignment relationship of image and content description text, that is, make multimodal large model have the ability of detecting object spatial position.On this basis, when multimodal large model is applied to open question answering task, it can accurately give correct answer based on the ability mastered when answering questions related to spatial arrangement.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and more specifically, to a method, apparatus and related equipment for training open-ended question answering and multimodal large models. Background Technology

[0002] With the development of artificial intelligence technology, especially the maturity of large-scale model technology, more and more domain-specific tasks are being handled through large-scale models. Taking image content understanding as an example, existing technologies can train multimodal large-scale models to understand image content and perform open-ended question answering based on user-input images.

[0003] Multimodal large models typically represent the input image as an embedding vector, and simultaneously convert the input question text into an embedding vector. Both vectors are then input into a Large Language Model (LLM), which decodes and outputs the answer text corresponding to the question. To achieve this, pre-training of the multimodal large model is generally required. During pre-training, image samples are represented as embeddings and input into the LLM, which decodes and outputs image description text. Training aims to align the LLM output with the description text labels corresponding to the image samples. This pre-training process ignores the spatial arrangement information of objects in the image, which can easily lead to spatial distortions in the trained multimodal large model, such as… Figure 1a and Figure 1b ,against Figure 1a The user poses questions to the existing multimodal large model, and the model outputs the following answers: Q1: What's on the table in front? A1: There's a potted plant and a small vase on the table in front. Q2: What's on the table behind? A2: There's a wooden coffee table on the table behind, with a small vase and a potted plant on it. (This is related to...) Figure 1b The user posed some questions to the existing multimodal large model, and the model output the following answers: Q1: What's on top of the sofa? A1: There's a bookshelf on top of the sofa, and various books are displayed on the bookshelf. Q2: What's to the left of the sofa? A2: There's a vase to the left of the sofa. These examples clearly show that the existing multimodal large model has failed to learn the spatial arrangement information of objects in the image, leading to spatially incorrect answers when dealing with questions related to spatial arrangement. Summary of the Invention

[0004] In view of the above problems, this application proposes to provide an open-ended question answering and multimodal large model training method, apparatus, and related equipment to solve the problem that existing multimodal large models cannot learn the spatial arrangement information of objects in images, and are prone to errors when answering questions related to spatial arrangement. The specific solution is as follows:

[0005] Firstly, it provides an open-ended question-and-answer method, including:

[0006] Obtain the input target image and the question to be asked;

[0007] The configured multimodal large model is invoked, and the target image and the question are input into the multimodal large model to obtain the output answer text corresponding to the question.

[0008] The multimodal large model is pre-trained using training images and matched image description text with spatial information, whereby the spatial information is used to represent the spatial location of objects in the training images.

[0009] Preferably, before invoking the configured multimodal large model, the method further includes: pre-training the multimodal large model, the pre-training process including:

[0010] Obtain the training image set;

[0011] Generate image description text with spatial information that matches the training images in the training image set, wherein the spatial information is used to represent the spatial location of the object in the training image;

[0012] Using the training images and the matching image description text, a multimodal large model is pre-trained until the set training termination condition is met to obtain the pre-trained multimodal large model.

[0013] Preferably, the spatial information in the image description text includes the coordinate information of the objects in the training image.

[0014] Preferably, the process of generating image description text with spatial information that matches the training images in the training image set includes:

[0015] Obtain the initial image description text of the training images in the training image set;

[0016] Obtain the coordinate information of the detection box where the object is located in the training image;

[0017] The coordinate information of the detection box containing the object is added to the initial image description text to obtain an image description text with spatial information that matches the training image.

[0018] Preferably, the process of obtaining the coordinate information of the detection box where the object is located in the training image includes:

[0019] In an open-source dataset for object detection, obtain the coordinate information of the detection bounding boxes of each object in the training image;

[0020] or,

[0021] Target detection is performed on the training image, and the coordinate information of the obtained object detection box is determined.

[0022] Preferably, before adding the coordinate information of the detection box containing the object to the initial image description text, the method further includes:

[0023] The coordinate information of the detection box where the object is located is normalized according to the size of the training image, and then multiplied by 10^n to obtain the normalized coordinate information, where n is a positive integer greater than or equal to 1.

[0024] Preferably, before invoking the configured multimodal large model, the method further includes: supervised fine-tuning of the pre-trained multimodal large model, wherein the supervised fine-tuning process includes:

[0025] Obtain supervised training data, which includes training images, spatial arrangement-related question texts posed for the training images, and matching answer tags, wherein the answer tags include coordinate information of objects in the training images;

[0026] Using the supervised training data, the pre-trained multimodal large model is fine-tuned to obtain the fine-tuned multimodal large model.

[0027] Preferably, the process of acquiring supervised training data includes:

[0028] In an open-source dataset for object detection, obtain the coordinate information of each detected object in the training image;

[0029] Based on the coordinate information of the object, determine the basic spatial orientation of the object in the training image, and / or, based on the coordinate information of each object, determine the spatial orientation relationship between different objects;

[0030] Based on the basic spatial orientation of each object and / or the spatial orientation relationship between different objects, a pre-configured question and answer template is used to generate question text related to spatial arrangement and matching answer tags.

[0031] Identify the object contained in the answer label and add the coordinate information of the object to the answer label.

[0032] Preferably, determining the basic spatial orientation of the object in the training image based on the object's coordinate information includes:

[0033] The training image is divided into multiple different directional regions, each directional region corresponding to a basic spatial orientation;

[0034] Based on the target orientation region to which the object's coordinate information belongs, the basic spatial orientation of the object in the training image is obtained.

[0035] Preferably, the process of acquiring supervised training data includes:

[0036] The initial descriptive text of the training images and the coordinate information of each object in the images are obtained based on the open-source dataset;

[0037] Obtain template prompt instructions, wherein the template prompt instructions include image information slots, and the template prompt instructions are used to instruct the large language model to design question-and-answer dialogue text based on the information in the image information slots and the spatial positional relationships of objects in the image;

[0038] The initial description text of the training image and the coordinate information of the objects in it are filled into the image information slot of the template prompt instruction to obtain the edited prompt instruction, which is then input into the configured large language model to obtain the question text and the matching answer label output by the model;

[0039] Identify the object contained in the answer label and add the coordinate information of the object to the answer label.

[0040] Preferably, after obtaining the output answer text corresponding to the question, the method further includes:

[0041] The answer text is post-processed to remove the coordinate information of objects from the answer text.

[0042] Secondly, a training method for multimodal large models is provided, including:

[0043] Obtain the training image set;

[0044] Generate image description text with spatial information that matches the training images in the training image set, wherein the spatial information is used to represent the spatial location of the object in the training image;

[0045] Using the training images and the matching image description text, a multimodal large model is pre-trained until the set training termination condition is met to obtain the pre-trained multimodal large model.

[0046] Thirdly, an open-ended question-and-answer device is provided, including:

[0047] The input acquisition unit is used to acquire the target image and the question being asked.

[0048] The multimodal large model invocation unit is used to invoke the configured multimodal large model, input the target image and the question into the multimodal large model, and obtain the output answer text corresponding to the question; wherein, the multimodal large model is pre-trained using training images and matched image description text with spatial information, and the spatial information is used to represent the spatial position of the object in the training image.

[0049] Fourthly, a training device for a multimodal large model is provided, comprising:

[0050] The training image set acquisition unit is used to acquire the training image set;

[0051] The image description text generation unit is used to generate image description text with spatial information that matches the training images in the training image set, wherein the spatial information is used to represent the spatial location of the object in the training image in the training image;

[0052] The pre-training unit is used to pre-train the multimodal large model using the training images and the matching image description text until the set training termination condition is met to obtain the pre-trained multimodal large model.

[0053] Fifthly, a data processing device is provided, comprising: a memory and a processor;

[0054] The memory is used to store programs;

[0055] The processor is configured to execute the program to implement the steps of the multimodal large model training method as described above, or to implement the steps of the open question answering method as described above.

[0056] In a sixth aspect, a storage medium is provided on which a computer program is stored, which, when executed by a processor, implements the steps of the multimodal large model training method as described above, or implements the steps of the open question answering method as described above.

[0057] By employing the above technical solution, this application, in order to encourage the multimodal large model to focus on spatial information, generates matching image description text with spatial information for training images during the pre-training stage. Compared to traditional training data, this embodiment adds explicit spatial information to the image description text corresponding to the training images. This spatial information represents the spatial location of objects contained in the training images; that is, it is a descriptive information about the spatial arrangement of objects in the training images. By pre-training the multimodal large model using training images and the aforementioned image description text with added explicit object spatial information, the multimodal large model can further focus on the spatial location of objects in the images, building upon its understanding of the semantic alignment between images and content description text. This enables the multimodal large model to detect the spatial location of objects. Based on this, when the multimodal large model is applied to open-ended question-answering tasks, it can accurately provide correct answers based on its acquired capabilities when answering questions related to spatial arrangement. Attached Figure Description

[0058] Various other advantages and benefits will become apparent to those skilled in the art upon reading the following detailed description of preferred embodiments. The accompanying drawings are for illustrative purposes only and are not intended to limit the scope of this application. Furthermore, the same reference numerals denote the same parts throughout the drawings. In the drawings:

[0059] Figures 1a-1b Two images illustrating visual questions were provided as examples.

[0060] Figure 2 An example is a schematic diagram of the structure of a multimodal large model;

[0061] Figure 3 A flowchart illustrating a training method for a multimodal large model provided in an embodiment of this application;

[0062] Figure 4 An example training image used to train a large multimodal model is provided;

[0063] Figure 5 An example is provided illustrating the directional region division of an image;

[0064] Figure 6 This application provides a schematic flowchart of an open-ended question-and-answer method.

[0065] Figure 7 This is a schematic diagram of a multimodal large model training device provided in an embodiment of this application;

[0066] Figure 8 A schematic diagram of an open-ended question-and-answer device provided in an embodiment of this application;

[0067] Figure 9 This is a schematic diagram of the structure of the data processing device provided in the embodiments of this application. Detailed Implementation

[0068] Before introducing the proposed solution, let's first explain the English terms used in this document:

[0069] prompt: Instructions. When interacting with AI (such as artificial intelligence models), you need to send instructions to the AI. These can be a text description, such as "Please recommend a popular song for me" when interacting with AI, or a parameter description in a certain format, such as describing the relevant drawing parameters to ask the AI ​​to draw a picture in a certain format.

[0070] Artificial intelligence models, also known as large-scale deep learning models, are artificial intelligence models based on deep learning technology. They consist of hundreds of millions of parameters and can perform complex tasks such as natural language processing, image recognition, and speech recognition through learning and training on massive amounts of data. Artificial intelligence models can include large language models and large-scale pre-trained models.

[0071] Large language models (LLMs) typically refer to language models with a large number of parameters and capabilities. They learn the statistical regularities and semantic relationships of language through pre-training on large-scale text data. These models often use unsupervised learning methods to predict the next word or fill in missing words to capture the context and semantic information of the language. Large language models can generate coherent sentences, answer questions, and perform translation tasks. A key characteristic of LLMs is their massive scale, containing billions or even more parameters, which help them learn complex patterns in language data. Emergent capabilities of large language models include context learning, instruction following, and progressive reasoning.

[0072] Multimodal large models refer to machine learning models that can simultaneously process multiple data types, such as text, images, audio, or video. These models are better able to capture the relationships between various data types and help improve the model's accuracy and efficiency. In the field of natural language processing, multimodal large models can be based on pre-trained models such as BERT, and fine-tuned by incorporating multimodal information such as audio and visual data to achieve more comprehensive and accurate analysis.

[0073] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0074] This application provides a training scheme for a multimodal large model, and the specific application of the trained multimodal large model in downstream related tasks.

[0075] The training process for large multimodal models can be implemented on servers, the cloud, or other terminals. The application process for large multimodal models can be implemented on application devices, such as user terminals, or through a combination of user terminals and servers.

[0076] There are various types of multimodal large model structures. This embodiment introduces one optional multimodal large model structure, combined with... Figure 2 As shown, a multimodal large model can include:

[0077] The VIT (Visual Transformer) module, the Q-Former (Querying Transformer) module, and the large language model.

[0078] The image is first encoded using the VIT module and then input into the Q-Former module to obtain the image feature representation.

[0079] The input to the large language model consists of two parts: one is the image feature representation output by the Q-Former module, and the other is the feature representation of the question text related to the image. The large language model decodes the input to obtain the output text.

[0080] Traditional approaches to pre-training multimodal large-scale models typically involve acquiring image samples and their standard descriptive texts, such as "There's a cat and a dog on the grass" or "A man is playing badminton on the playground." During pre-training, the image samples are fed into the VIT module, and after passing through the Q-Former module, image feature representations are obtained. These image feature representations are then used as implicit prompts and fed into the large language model. The large language model uses the standard descriptive text corresponding to the image samples as labels and calculates the loss of the output text, maximizing the likelihood probability of the output image description text. The loss function can typically be expressed as:

[0081]

[0082] Where s represents the image feature representation obtained by the Q-Former module, and wi This represents the i-th token in the output image description text, where N represents the total length of the image description text. During the pre-training phase, the network parameters of both the Q-Former module and the large language model can be optimized simultaneously.

[0083] When using the aforementioned image samples and their conventional descriptive text to pre-train a multimodal large model, the multimodal large model mainly focuses on the perception and understanding of image content, that is, semantically aligning the image with the content description text. This training process ignores the spatial arrangement information of objects in the image, which leads to some spatial disorder problems in the pre-trained multimodal large model.

[0084] Therefore, this application provides a training scheme for a multimodal large model to enable the multimodal large model to pay attention to the spatial information of objects in an image. The following will combine... Figure 3 The multimodal large model training method described in this application may include the following steps:

[0085] Step S100: Obtain the training image set.

[0086] Specifically, a large number of training images can be obtained through various means to form a training image set, such as obtaining training images from open-source image datasets. The training images can include one or more different entity objects.

[0087] Step S110: Generate image description text with spatial information that matches the training images in the training image set, wherein the spatial information is used to represent the spatial location of the object in the training image.

[0088] It's important to note that traditional multimodal large model training methods are limited to semantically aligning images with their descriptive text. Consequently, the traditional descriptive text for training images is also limited to describing which objects are present in the image, rarely or never including the spatial locations of those objects within the training image. When using this type of data for pre-training a multimodal large model, the model struggles to learn the spatial arrangement information of objects within the training images.

[0089] To address the aforementioned deficiencies in the prior art, this application generates image description texts carrying spatial information for each training image in the training image set, which are matched with the training images. The spatial information is a description of the spatial arrangement of objects in the training images. This results in a large number of image description texts carrying spatial information and their corresponding training images, forming pre-training data.

[0090] The spatial information of objects in the descriptive text of training images can be described in various formats, such as the coordinate information of the object in the training image.

[0091] Step S120: Using the training images and the matched image description text, pre-train the multimodal large model until the set training termination condition is met to obtain the pre-trained multimodal large model.

[0092] Adding image description text with spatial information during pre-training allows the multimodal large model to focus on the spatial location information of each object while understanding the image content, thereby avoiding the loss of spatial information.

[0093] The multimodal large model training method provided in this application, in order to encourage the multimodal large model to pay attention to spatial information, generates matching image description text with spatial information for training images during the pre-training stage. Compared with traditional training data, this embodiment adds explicit spatial information to the image description text corresponding to the training images. The spatial information is used to represent the spatial position of objects contained in the training images, that is, the spatial information is a description of the spatial arrangement of objects in the training images. By using the training images and the aforementioned image description text with added explicit object spatial information to pre-train the multimodal large model, the multimodal large model can further focus on the spatial position of objects in the images based on learning the semantic alignment relationship between images and content description text, thus enabling the multimodal large model to have the ability to detect the spatial position of objects. On this basis, when the multimodal large model is applied to open-ended question answering tasks, it can accurately give correct answers based on its mastery of the ability to answer questions related to spatial arrangement.

[0094] Combination Figure 4 For example, in the training image, the conventional image description text in the prior art is "A man is holding a tennis racket on a court, and dust is flying everywhere." It can be seen that the conventional image description text only describes the semantic content of the image and does not include the spatial information of the object.

[0095] When using the scheme of this application, an image description text with spatial information that matches the training image is generated. Here, coordinate information with spatial information as the object is used as an example for explanation. The image description text carrying spatial information can be "A man (coordinates: [0.42, 0.22, 0.62, 0.94]) is holding a tennis racket on the court (coordinates: [0.32, 0.34, 0.47, 0.51]), and dust is flying on the court."

[0096] Specifically, the object's coordinate information can be the x and y coordinates of the top-left corner and the bottom-right corner of the detection box where the object is located. Of course, other coordinate representation methods can also be used, which will not be elaborated here.

[0097] It should be noted that the coordinate information of objects in the image description text can be inserted after the object, or the coordinate information of all objects can be inserted after the regular image description text. For example, "A man is holding a tennis racket on a court, and dust is flying everywhere. (Man's coordinates: [0.42, 0.22, 0.62, 0.94], Tennis racket coordinates: [0.32, 0.34, 0.47, 0.51])".

[0098] Furthermore, spatial information can be formatted not only using coordinates but also other formats, such as using spatial directional terms. Figure 4 Taking the training image as an example, the image description text carrying spatial information could be "A man (located in the center of the image) is holding a tennis racket on a court (located to the left of the man), with dust flying around on the court."

[0099] In some embodiments of this application, coordinate information, which is spatial information, is used as an example for illustration.

[0100] The above step S110, the process of generating image description text with spatial information that matches the training images in the training image set, may include:

[0101] S1. Obtain the initial image description text of the training images in the training image set.

[0102] Specifically, an open-source dataset can be used to obtain the initial image description text of the training images in the training image set. The initial image description text can be understood as the aforementioned conventional image description text, which does not carry spatial information of the object.

[0103] S2. Obtain the coordinate information of the detection box where the object is located in the training image.

[0104] Specifically, the training image set can be an open-source dataset for object detection tasks, in which each training image is labeled with an object detection box, and the position coordinates of each object detection box are known, that is, the coordinate information of the object can be obtained.

[0105] In addition, other open-source datasets can be used as training image sets. Object detection algorithms can be applied to the training images to determine the coordinate information of the object detection boxes, that is, to obtain the coordinate information of the objects in the training images.

[0106] S3. Add the coordinate information of the detection box where the object is located to the initial image description text to obtain an image description text with spatial information that matches the training image.

[0107] Specifically, after obtaining the coordinate information of the detection box where the object is located in the training image (that is, the coordinate information of the object), this coordinate information is the spatial information of the object, which can be added to the initial image description text to obtain an image description text with spatial information that matches the training image.

[0108] As mentioned above Figure 4 Example training images:

[0109] First, obtain the initial image description text that matches it: "A man is holding a tennis racket on a court, and dust is flying everywhere."

[0110] The training image was subjected to object detection to obtain the coordinate information of each object contained therein, namely: male coordinates: [0.42, 0.22, 0.62, 0.94], tennis racket coordinates: [0.32, 0.34, 0.47, 0.51].

[0111] Furthermore, according to the set addition rules, the coordinate information of the object is added to the initial image description text. The set addition rules may include adding the coordinate information of the object to the object. The final image description text with spatial information is: "A man (coordinates: [0.42, 0.22, 0.62, 0.94]) is holding a tennis racket on the court (coordinates: [0.32, 0.34, 0.47, 0.51]), and dust is flying on the court."

[0112] Optionally, considering the coordinate information in the above example, the coordinate information of the object obtained through target detection may contain the same bytes, such as the first two bytes of each coordinate value being "0" and ".". These identical bytes do not help the model learn the spatial location of the object and will occupy additional storage space. Therefore, in this embodiment, before adding the coordinate information of the detection box where the object is located to the initial image description text, a preprocessing process for the coordinate information can be added, and then the preprocessed coordinate information can be added to the initial image description text. The preprocessing process for the coordinate information may include:

[0113] The coordinates of the detection box containing the object are normalized to the size of the training image, then multiplied by 10^n to obtain the normalized coordinates, where n is a positive integer greater than or equal to 1. For example, n can be 2. After adding the normalized coordinates to the initial image description text, the result is as follows:

[0114] “A man (coordinates: [42, 22, 62, 94]) is holding a tennis racket on the court (coordinates: [32, 34, 47, 51]), and dust is flying around the court.”

[0115] Through the pre-training process of the multimodal large model exemplified in the above embodiments, the multimodal large model can learn to detect the location of objects in an image, that is, it has spatial awareness. This pre-trained multimodal large model can be used as a base model for optimization in downstream tasks.

[0116] This embodiment illustrates some optional downstream task scenarios for multimodal large models, including but not limited to:

[0117] 1) In general scenarios, spatially related visual question answering can better handle user questions about space. For example, a user can ask a question about an image: "What is on the table on the right side of the image?"

[0118] 2) Perform descriptive image target detection. Provide detailed attribute descriptions for the items to be detected, such as "detect the person holding dangerous items in the image" or "detect the person who has fallen in the image".

[0119] 3) In intelligent robot scenarios, multimodal large models can help intelligent robots achieve better spatial perception. For example, in home scenarios, when receiving spatial-related instructions from users such as "open the cabinet on the left", they can better understand the user's intention.

[0120] This application embodiment takes the first type of downstream task scenario in the above example as an example for further explanation.

[0121] In real-world scenarios, users often don't directly ask questions about specific coordinates. Questions related to spatial relationships are typically directional, such as "What's to the left / right of the image?", "What's in the center of the image?", "What's to the left / right of an object?", "What's on the table in front?", etc. To associate object coordinates with these more linguistic spatial directional terms, this embodiment can further construct supervised training data. This supervised training data can then be used to fine-tune the pre-trained multimodal large model, resulting in a fine-tuned multimodal large model.

[0122] The supervised training data may include training images, spatial arrangement-related question texts posed to the training images, and matching answer labels, wherein the answer labels include the coordinate information of objects in the training images.

[0123] It is understandable that, since the question text is related to spatial arrangement, such as the various types of orientation questions in the aforementioned examples, spatial orientation terms will be included in the question text and / or answer tags. In this embodiment, in order to associate the coordinate information represented by specific numerical values ​​with the linguistic spatial orientation terms, the coordinate information of the object is further added to the answer tags, that is, supervised training data containing spatial orientation terms and object coordinate information is constructed.

[0124] by Figure 1b Taking the training image shown as an example, the constructed supervised training data includes, but is not limited to:

[0125] Q1: Where is the bookshelf located in the diagram? A1: The bookshelf (coordinates: [0.85, 0.36, 0.98, 0.70]) is on the far right of the diagram.

[0126] Q2: What is on the far right of the picture? A2: The far right of the picture is a bookshelf (coordinates: [0.85, 0.36, 0.98, 0.70]).

[0127] Q3: What's to the left of the sofa? A3: There's a floor lamp to the left of the sofa (coordinates: [0.08, 0.60, 0.92, 0.75]) (coordinates: [0.00, 0.40, 0.10, 0.68]).

[0128] As can be seen from the supervised training data in the above examples, the question texts are all related to spatial arrangement. Therefore, the question texts and answer labels will contain spatial orientation words and object coordinate information.

[0129] After fine-tuning the pre-trained multimodal large model using the aforementioned supervised training data, the multimodal large model can learn the relationship between coordinate information and spatial location words, thereby better understanding the spatial location questions raised by users and providing accurate answer text.

[0130] In some embodiments of this application, several optional methods for constructing the supervised training data described above are introduced.

[0131] The first construction method may include the following steps:

[0132] S11. In the open-source dataset for the object detection task, obtain the coordinate information of each detected object in the training image.

[0133] Specifically, open-source datasets for object detection tasks contain training images labeled with the coordinates of object detection boxes. The coordinate information of the detection boxes of each object in the training images can be obtained as the object's coordinate information.

[0134] S12. Based on the coordinate information of the object, determine the basic spatial orientation of the object in the training image, and / or, based on the coordinate information of each object, determine the spatial orientation relationship between different objects.

[0135] Specifically, in this step, the basic spatial orientation of the object can be determined based on the object's coordinate information, such as front, back, left, right, up, down, etc.

[0136] This embodiment provides an optional implementation method, combined with Figure 5 As shown, the training image can be divided into multiple different directional regions, each corresponding to a basic spatial orientation, as detailed below. Figure 5 As shown. Of course, Figure 5 This example only illustrates one possible method for dividing the directional area; other methods can be selected as needed.

[0137] Based on this, the target orientation region to which the object belongs is determined according to the coordinate information of the object in the training image, thus obtaining the basic spatial orientation of the object in the training image.

[0138] Exemplary, known Figure 5 The objects included are: sofa, floor lamp, and bookshelf, with their respective coordinate information as follows:

[0139] Sofa: [0.08, 0.60, 0.92, 0.75]

[0140] Floor lamp: [0.00, 0.40, 0.10, 0.68]

[0141] Bookshelf: [0.85, 0.36, 0.98, 0.70].

[0142] In the example above, the object's coordinates are the coordinates of the top-left and bottom-right vertices of the detection box where the object is located. Furthermore, based on the object's coordinates within the detection box, it can be determined which target orientation region it belongs to.

[0143] Taking a bookshelf as an example, the coordinates of the center point of its detection frame [0.91, 0.53] can be calculated. It is determined that the coordinates of the center point fall into the rightmost region, thus the target orientation region to which the bookshelf belongs is determined to be: the rightmost region.

[0144] Of course, the above only illustrates the case of determining the target orientation region of an object based on the positional relationship between the center point coordinates of the detection box and each basic orientation region. For example, the basic orientation region with the largest overlapping area is selected as the target orientation region of the object according to the size of the overlapping area between the detection box of the object and each basic orientation region.

[0145] In this step, the spatial orientation relationship between different objects can also be determined by comparing the coordinate information of different objects in the same training image. For example, in which direction is object a relative to object b?

[0146] Still with Figure 5 Taking this as an example, based on the coordinates of different objects, we can see that the floor lamp is located to the left of the sofa.

[0147] S13. Based on the basic spatial orientation of each object and / or the spatial orientation relationship between different objects, use a pre-configured question and answer template to generate question text and matching answer tags related to spatial arrangement.

[0148] Specifically, there can be various question-and-answer templates. For example, the first type can be a template for asking about the spatial location of an object, the second type can be a template for asking about the spatial relationship between different objects, and so on.

[0149] Example of a first type of template:

[0150] Question template 1: "Where is x1 located in the diagram?", where x1 is the object to be added. The corresponding answer tag template is: "x1 is located at y1 in the diagram", where y1 is the basic spatial orientation of x1 to be added.

[0151] Based on the above Figure 5 In the example, x1 can be "bookshelf" and the corresponding y1 is "the rightmost".

[0152] Question template 2: "What is y1 in the figure?", the corresponding answer tag template is: "y1 in the figure is x1", where the definitions of x1 and y1 are the same as before.

[0153] Example of the second type of template:

[0154] Question template 1: "What is to the left of x1?", the corresponding answer label template is: "x2 is to the left of x1", where x1 and x2 are different objects, and their positional relationship can be determined based on the pre-determined coordinates of the different objects. The word "to the left" in both the question and answer label templates can be replaced with other directional terms, depending on the relative positional relationship between the two objects in the training image.

[0155] S14. Determine the objects contained in the answer labels and add the coordinate information of the objects to the answer labels.

[0156] Specifically, in step S11 above, the position coordinates of each object in the training image are determined. Depending on the question, the objects included in the answer label may also be different. In this step, it is necessary to determine the objects included in the answer label and then add the coordinate information of the objects to the answer label.

[0157] by Figure 5Taking the training image shown as an example, for the generated question text "Where is the sofa located in the image?", the corresponding answer label is "The sofa is on the far right of the image". Since the training image contains three objects: a sofa, a floor lamp, and a bookshelf, we first determine the object contained in the generated question text, namely "sofa". Then, we can add the sofa's coordinates to the generated answer label, resulting in the edited answer label "The sofa (coordinates: [0.08, 0.60, 0.92, 0.75]) is on the far right of the image". The generated question text and the edited answer label together constitute a supervised training dataset.

[0158] It is understandable that the coordinate information can be preprocessed before adding the object's coordinate information to the answer label. The preprocessing process is the same as described in the previous embodiment, and you can refer to the relevant description in the previous text for details.

[0159] The above embodiments introduce a process for constructing supervised training data, which can determine the basic spatial orientation of objects such as up, down, left, and right based on the coordinate information of objects in the training image, and then generate question and answer data related to the basic orientation according to the question and answer template.

[0160] In some embodiments of this application, a second method for constructing supervised training data is described, which may specifically include the following steps:

[0161] S21. Based on the open-source dataset, obtain the initial description text of the training images and the coordinate information of each object in the images.

[0162] S22. Obtain template prompt instructions.

[0163] The template prompt instruction includes an image information slot. The template prompt instruction is used to instruct the large language model to design question-and-answer dialogue text based on the information in the image information slot and the spatial relationship of objects in the image.

[0164] An optional template prompt is as follows: Based on the image content description and the coordinate information of some objects {image information slots}, design some natural question-and-answer dialogues regarding the spatial relationships of the objects in the image.

[0165] S23. Fill the image information slot of the template prompt instruction with the initial description text of the obtained training image and the coordinate information of the objects in it to obtain the edited prompt instruction, and input it into the configured large language model to obtain the question text and the matching answer label output by the model.

[0166] by Figure 1a Let's take an example to illustrate:

[0167] against Figure 1aThe training image shown has the following image information (including the initial description text and the coordinate information of each object in the image):

[0168] Initial description: This image shows an indoor relaxation corner. There are two round wooden coffee tables, one with an open book and a cup on it. On the other coffee table is a glass bottle with some branches and leaves.

[0169] Table: [0.05, 0.55, 0.48, 0.85], [0.24, 0.59, 0.92, 0.97]

[0170] Book: [0.42, 0.62, 0.80, 0.67]

[0171] Coffee cup: [0.80, 0.62, 0.83, 0.65].

[0172] The image information obtained above is filled into the image information slot of the template prompt instruction to obtain the edited prompt instruction, which is then fed into the large language model to obtain the question-and-answer dialogue output by the large language model as follows:

[0173] Q: What's on the table on the right?

[0174] A: There is a book and a coffee cup on the table to the right.

[0175] Q: What's on the table on the left?

[0176] A: There is a glass on the table to the left.

[0177] ...

[0178] S24. Determine the objects contained in the answer labels and add the coordinate information of the objects to the answer labels.

[0179] This step is similar to step S14 mentioned above. For detailed implementation process, please refer to the previous description. It will not be repeated here.

[0180] The method provided in this embodiment can leverage the semantic understanding and text generation capabilities of large language models to generate more open and richer spatial question-and-answer data.

[0181] Based on the training methods for the multimodal large model described in the foregoing embodiments, a trained multimodal large model can be obtained. This trained model possesses spatial awareness capabilities. Building upon this, this embodiment introduces an open-ended question-answering method based on the trained multimodal large model, referring to... Figure 6 Open-ended question-and-answer methods can specifically include:

[0182] Step S200: Obtain the input target image and the question.

[0183] The target image is the image specified by the user for visual question-and-answer purposes. The question is the question posed by the user to the target image, which can be in audio or text format. Audio questions can be further converted into text format.

[0184] Step S210: Call the configured multimodal large model, wherein the multimodal large model is pre-trained using training images and matched image description text with spatial information, wherein the spatial information is used to represent the spatial location of objects in the training images.

[0185] Specifically, the multimodal large model called in this step can be a multimodal large model trained using the training method described in any of the foregoing embodiments. After the above training, the multimodal large model possesses spatial awareness capabilities, thus enabling it to better handle user-generated questions related to spatial arrangement and accurately provide answer text.

[0186] Step S220: Input the target image and the question into the multimodal large model to obtain the output answer text corresponding to the question.

[0187] It should be noted that in the supervised training process of the multimodal large model described in the aforementioned embodiments, the coordinate information of the object is added to the answer label of the training data. Therefore, the answer text output by the multimodal large model corresponding to the image and text question may also contain the coordinate information of the object. If it is not necessary to show the coordinate information of the object to the user, the generated answer text can be post-processed before being output to the user to remove the coordinate information of the object in the answer text.

[0188] The training apparatus for a multimodal large model provided in the embodiments of this application will be described below. The training apparatus for a multimodal large model described below can be referred to in correspondence with the training method for a multimodal large model described above.

[0189] See Figure 7 , Figure 7 This is a schematic diagram of a training device for a multimodal large model disclosed in an embodiment of this application.

[0190] like Figure 7 As shown, the device may include:

[0191] Training image set acquisition unit 11 is used to acquire training image set;

[0192] Image description text generation unit 12 is used to generate image description text with spatial information that matches the training images in the training image set, wherein the spatial information is used to represent the spatial location of the object in the training image in the training image;

[0193] The pre-training unit 13 is used to pre-train the multimodal large model using the training images and the matched image description text until the set training termination condition is met to obtain the pre-trained multimodal large model.

[0194] Optionally, the spatial information in the image description text generated by the image description text generation unit above includes the coordinate information of the objects in the training image.

[0195] Optionally, the process by which the image description text generation unit generates image description text with spatial information that matches the training images in the training image set includes:

[0196] Obtain the initial image description text of the training images in the training image set;

[0197] Obtain the coordinate information of the detection box where the object is located in the training image;

[0198] The coordinate information of the detection box containing the object is added to the initial image description text to obtain an image description text with spatial information that matches the training image.

[0199] Optionally, the process by which the image description text generation unit obtains the coordinate information of the detection box where the object is located in the training image includes:

[0200] In an open-source dataset for object detection, obtain the coordinate information of the detection bounding boxes of each object in the training image;

[0201] or,

[0202] Target detection is performed on the training image, and the coordinate information of the obtained object detection box is determined.

[0203] Optionally, before adding the coordinate information of the detection box where the object is located to the initial image description text, the image description text generation unit further includes:

[0204] The coordinate information of the detection box where the object is located is normalized according to the size of the training image, and then multiplied by 10^n to obtain the normalized coordinate information, where n is a positive integer greater than or equal to 1.

[0205] Optionally, the apparatus of this application may further include:

[0206] A supervised training data acquisition unit is used to acquire supervised training data, which includes training images, spatial arrangement-related question texts proposed for the training images, and matching answer tags. The answer tags include coordinate information of objects in the training images.

[0207] A supervised training unit is used to fine-tune the pre-trained multimodal large model using the supervised training data to obtain a fine-tuned multimodal large model.

[0208] Optionally, the process by which the supervised training data acquisition unit acquires supervised training data includes:

[0209] In an open-source dataset for object detection, obtain the coordinate information of each detected object in the training image;

[0210] Based on the coordinate information of the object, determine the basic spatial orientation of the object in the training image, and / or, based on the coordinate information of each object, determine the spatial orientation relationship between different objects;

[0211] Based on the basic spatial orientation of each object and / or the spatial orientation relationship between different objects, a pre-configured question and answer template is used to generate question text related to spatial arrangement and matching answer tags.

[0212] Identify the object contained in the answer label and add the coordinate information of the object to the answer label.

[0213] Optionally, the process by which the supervised training data acquisition unit determines the basic spatial orientation of the object in the training image based on the object's coordinate information includes:

[0214] The training image is divided into multiple different directional regions, each directional region corresponding to a basic spatial orientation;

[0215] Based on the target orientation region to which the object's coordinate information belongs, the basic spatial orientation of the object in the training image is obtained.

[0216] In another optional scenario, the process by which the supervised training data acquisition unit acquires supervised training data may include:

[0217] The initial descriptive text of the training images and the coordinate information of each object in the images are obtained based on the open-source dataset;

[0218] Obtain template prompt instructions, wherein the template prompt instructions include image information slots, and the template prompt instructions are used to instruct the large language model to design question-and-answer dialogue text based on the information in the image information slots and the spatial positional relationships of objects in the image;

[0219] The initial description text of the training image and the coordinate information of the objects in it are filled into the image information slot of the template prompt instruction to obtain the edited prompt instruction, which is then input into the configured large language model to obtain the question text and the matching answer label output by the model;

[0220] Identify the object contained in the answer label and add the coordinate information of the object to the answer label.

[0221] Some embodiments of this application further disclose an open-ended question-answering device, which can be referred to in correspondence with the open-ended question-answering method described above.

[0222] See Figure 8 , Figure 8 This is a schematic diagram of an open-ended question-and-answer device disclosed in an embodiment of this application.

[0223] like Figure 8 As shown, the device may include:

[0224] The input acquisition unit 21 is used to acquire the input target image and the question asked.

[0225] The multimodal large model invocation unit 22 is used to invoke the configured multimodal large model, input the target image and the question into the multimodal large model, and obtain the output answer text corresponding to the question. The multimodal large model is pre-trained using training images and matched image description text with spatial information. The spatial information is used to represent the spatial position of the object in the training image.

[0226] Further optionally, the open question-answering device may also include a multimodal large model training unit, used to train the multimodal large model before calling the configured multimodal large model. The specific training process can be referred to the relevant description in the foregoing embodiments, and will not be repeated here.

[0227] Optionally, the open-ended question-answering device of this application may also include:

[0228] If the multimodal large model being called has object coordinate information added to the training data during the supervised training process, the post-processing unit can further process the answer text after obtaining the output answer text corresponding to the question to remove the object coordinate information from the answer text.

[0229] The multimodal large model training device and open-ended question-answering device provided in this application embodiment can be applied to data processing equipment, such as servers, terminals, and cloud computing. Optionally, Figure 9 The hardware structure block diagram of the data processing device is shown, with reference to... Figure 9 The hardware structure of the device may include: at least one processor 1, at least one communication interface 2, at least one memory 3, and at least one communication bus 4;

[0230] In this embodiment of the application, the number of processor 1, communication interface 2, memory 3, and communication bus 4 is at least one, and processor 1, communication interface 2, and memory 3 communicate with each other through communication bus 4;

[0231] Processor 1 may be a central processing unit (CPU), an application-specific integrated circuit (ASIC), or one or more integrated circuits configured to implement embodiments of the present invention.

[0232] Memory 3 may include high-speed RAM, and may also include non-volatile memory, such as at least one disk storage device;

[0233] The memory stores a program, which the processor can call. The program is used to: implement the various steps of the aforementioned multimodal large model training method, or implement the various steps of the aforementioned open question answering method.

[0234] This application embodiment also provides a storage medium that can store a program suitable for execution by a processor. The program is used to: implement the various steps of the aforementioned training method for multimodal large models, or to implement the various steps of the aforementioned open question answering method.

[0235] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0236] The various embodiments in this specification are described in a progressive manner. Each embodiment focuses on the differences from other embodiments. The various embodiments can be combined as needed, and the same or similar parts can be referred to each other.

[0237] The above description of the disclosed embodiments enables those skilled in the art to make or use this application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this application. Therefore, this application is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. An open-ended question-and-answer method, characterized in that, include: Obtain the input target image and the question to be asked; The configured multimodal large model is invoked, and the target image and the question are input into the multimodal large model to obtain the output answer text corresponding to the question. The multimodal large model is pre-trained using training images and matched image description text with spatial information, wherein the spatial information is used to represent the spatial location of objects in the training images. Before invoking the configured multimodal large model, the process also includes: supervised fine-tuning of the pre-trained multimodal large model, which includes: Obtain supervised training data, which includes training images, question text related to spatial arrangement proposed for the training images, and matching answer tags. The answer tags include coordinate information of objects in the training images. The supervised training data includes spatial orientation words and coordinate information of objects. Using the supervised training data, the pre-trained multimodal large model is fine-tuned to obtain a fine-tuned multimodal large model, so that the fine-tuned multimodal large model can learn the correlation between coordinate information and spatial location words.

2. The method according to claim 1, characterized in that, Before invoking the configured multimodal large model, the following steps are included: pre-training the multimodal large model, which involves: Obtain the training image set; Generate image description text with spatial information that matches the training images in the training image set, wherein the spatial information is used to represent the spatial location of the object in the training image; Using the training images and the matching image description text, a multimodal large model is pre-trained until the set training termination condition is met to obtain the pre-trained multimodal large model.

3. The method according to claim 2, characterized in that, The spatial information in the image description text includes the coordinate information of objects in the training image.

4. The method according to claim 3, characterized in that, The process of generating image description text with spatial information that matches training images in the training image set includes: Obtain the initial image description text of the training images in the training image set; Obtain the coordinate information of the detection box where the object is located in the training image; The coordinate information of the detection box containing the object is added to the initial image description text to obtain an image description text with spatial information that matches the training image.

5. The method according to claim 4, characterized in that, The process of obtaining the coordinate information of the detection box where the object is located in the training image includes: In an open-source dataset for object detection, obtain the coordinate information of the detection bounding boxes of each object in the training image; or, Target detection is performed on the training image, and the coordinate information of the obtained object detection box is determined.

6. The method according to claim 4, characterized in that, Before adding the coordinate information of the detection box containing the object to the initial image description text, the method further includes: The coordinate information of the detection box where the object is located is normalized according to the size of the training image, and then multiplied by 10^n to obtain the normalized coordinate information, where n is a positive integer greater than or equal to 1.

7. The method according to claim 1, characterized in that, The process of acquiring supervised training data includes: In an open-source dataset for object detection, obtain the coordinate information of each detected object in the training image; Based on the coordinate information of the object, determine the basic spatial orientation of the object in the training image, and / or, based on the coordinate information of each object, determine the spatial orientation relationship between different objects; Based on the basic spatial orientation of each object and / or the spatial orientation relationship between different objects, a pre-configured question and answer template is used to generate question text related to spatial arrangement and matching answer tags. Identify the object contained in the answer label and add the coordinate information of the object to the answer label.

8. The method according to claim 7, characterized in that, Based on the coordinate information of the object, determining the basic spatial orientation of the object in the training image includes: The training image is divided into multiple different directional regions, each directional region corresponding to a basic spatial orientation; Based on the target orientation region to which the object's coordinate information belongs, the basic spatial orientation of the object in the training image is obtained.

9. The method according to claim 1, characterized in that, The process of acquiring supervised training data includes: The initial descriptive text of the training images and the coordinate information of each object in the images are obtained based on the open-source dataset; Obtain template prompt instructions, wherein the template prompt instructions include image information slots, and the template prompt instructions are used to instruct the large language model to design question-and-answer dialogue text based on the information in the image information slots and the spatial positional relationships of objects in the image; The initial description text of the training image and the coordinate information of the objects in it are filled into the image information slot of the template prompt instruction to obtain the edited prompt instruction, which is then input into the configured large language model to obtain the question text and the matching answer label output by the model; Identify the object contained in the answer label and add the coordinate information of the object to the answer label.

10. The method according to claim 1, characterized in that, After obtaining the output answer text corresponding to the question, the following is also included: The answer text is post-processed to remove the coordinate information of objects from the answer text.

11. A training method for a multimodal large model, characterized in that, include: Obtain the training image set; Generate image description text with spatial information that matches the training images in the training image set, wherein the spatial information is used to represent the spatial location of the object in the training image; Using the training images and the matched image description text, a multimodal large model is pre-trained until the set training termination condition is met to obtain the pre-trained multimodal large model. It also includes: supervised fine-tuning of the pre-trained multimodal large model, the supervised fine-tuning process including: Obtain supervised training data, which includes training images, question text related to spatial arrangement proposed for the training images, and matching answer tags. The answer tags include coordinate information of objects in the training images. The supervised training data includes spatial orientation words and coordinate information of objects. Using the supervised training data, the pre-trained multimodal large model is fine-tuned to obtain a fine-tuned multimodal large model, so that the fine-tuned multimodal large model can learn the correlation between coordinate information and spatial location words.

12. An open-ended question-and-answer device, characterized in that, include: The input acquisition unit is used to acquire the target image and the question being asked. The multimodal large model invocation unit is used to invoke the configured multimodal large model, input the target image and the question into the multimodal large model, and obtain the output answer text corresponding to the question; The multimodal large model is pre-trained using training images and matched image description text with spatial information, wherein the spatial information is used to represent the spatial location of objects in the training images. The device is further configured to: perform supervised fine-tuning on the pre-trained multimodal large model, the supervised fine-tuning process including: Obtain supervised training data, which includes training images, question text related to spatial arrangement proposed for the training images, and matching answer tags. The answer tags include coordinate information of objects in the training images. The supervised training data includes spatial orientation words and coordinate information of objects. Using the supervised training data, the pre-trained multimodal large model is fine-tuned to obtain a fine-tuned multimodal large model, so that the fine-tuned multimodal large model can learn the correlation between coordinate information and spatial location words.

13. A training device for a multimodal large model, characterized in that, include: The training image set acquisition unit is used to acquire the training image set; The image description text generation unit is used to generate image description text with spatial information that matches the training images in the training image set, wherein the spatial information is used to represent the spatial location of the object in the training image in the training image; The pre-training unit is used to pre-train the multimodal large model using the training images and the matched image description text until the set training termination condition is met to obtain the pre-trained multimodal large model. The device is further configured to: perform supervised fine-tuning on the pre-trained multimodal large model, the supervised fine-tuning process including: Obtain supervised training data, which includes training images, question text related to spatial arrangement proposed for the training images, and matching answer tags. The answer tags include coordinate information of objects in the training images. The supervised training data includes spatial orientation words and coordinate information of objects. Using the supervised training data, the pre-trained multimodal large model is fine-tuned to obtain a fine-tuned multimodal large model, so that the fine-tuned multimodal large model can learn the correlation between coordinate information and spatial location words.

14. A data processing device, characterized in that, include: Memory and processor; The memory is used to store programs; The processor is configured to execute the program to implement the steps of the open-ended question-answering method as claimed in any one of claims 1-10, or to implement the steps of the training method for the multimodal large model as claimed in claim 11.

15. A storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the steps of the open-ended question-answering method as claimed in any one of claims 1-10, or implements the steps of the training method for the multimodal large model as claimed in claim 11.

Images