Image-text pair construction method and device, computer device, and storage medium

Abstract

This invention relates to the field of artificial intelligence technology and discloses a method, apparatus, computer device, and storage medium for constructing image-text pairs. The method includes: acquiring a question set in a target scene; the question set includes a perception question set, a prediction question set, and a decision question set; constructing question prompts corresponding to the question set; inputting the perception-prediction question prompts and preset sample images into a first visual language model to generate perception-prediction information for the sample images; using the perception-prediction information as context, inputting a third question prompt and the sample images into a second visual language model to generate decision information for the sample images; and generating descriptive text for the sample images based on the perception-prediction information and the decision information, thus constructing an image-text pair containing the sample images and descriptive text. This invention, through a top-down prompt design of perception, prediction, and decision, can generate descriptive text containing rich details, thereby improving the quality of image-text pairs.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and specifically to a method, apparatus, computer device, and storage medium for constructing image-text pairs. Background Technology

[0002] In scenarios such as autonomous driving and robot control, it is necessary to acquire images of the surrounding environment using cameras and combine these images to achieve automatic control. However, traditional image recognition models have limitations; taking autonomous driving as an example, existing autonomous driving systems still have limitations in understanding complex traffic scenarios and predicting the intentions of vehicles and pedestrians.

[0003] The emergence of the Vision Language Model (VLM) marks a new stage in the development of autonomous driving technology. Relying on advanced large language models, VLM has outstanding performance in complex scene understanding and causal reasoning, demonstrating its great potential to handle complex situations, identify driving intentions, and make informed decisions in real driving environments.

[0004] Training a Visual Language Model (VLM) relies on a large corpus of aligned images, videos, and text. These aligned corpora are image-text pairs, or simply image-text pairs. To enable the VLM to accurately understand complex scenarios such as autonomous driving and robot control, it is necessary to construct a corpus containing a large number of text pairs for training the VLM. Summary of the Invention

[0005] In view of this, the present invention provides a method, apparatus, computer device and storage medium for constructing image-text pairs, so as to construct image-text pair corpora suitable for scenarios such as autonomous driving.

[0006] In a first aspect, the present invention provides a method for constructing image-text pairs, comprising:

[0007] Obtain a set of questions in the target scenario; the set of questions includes a set of perception questions, a set of prediction questions, and a set of decision questions.

[0008] Construct question prompts corresponding to the question set; the question prompts include a first question prompt corresponding to the perception question set, a second question prompt corresponding to the prediction question set, and a third question prompt corresponding to the decision question set;

[0009] The perceptual prediction question prompts and preset sample images are input into a first visual language model to generate perceptual prediction information for the sample images; the perceptual prediction question prompts include the first question prompt and / or the second question prompt.

[0010] Using the perceived prediction information as context, the third question prompt and the sample image are input into the second visual language model to generate decision information for the sample image;

[0011] Based on the perception prediction information and the decision information, a descriptive text for the sample image is generated, and an image-text pair containing the sample image and the descriptive text is constructed.

[0012] In some optional implementations, the step of constructing the question prompt words corresponding to the question set includes:

[0013] For each question in the question set, a format template corresponding to the question is constructed; the format template is a key-value pair template, and the key in the format template includes the key information of the question, and the value in the format template is used to represent the answer to the question;

[0014] Based on the question and the corresponding format template, generate format prompts; the format prompts indicate that the answer to the question should be generated according to the corresponding format template.

[0015] Based on the specified format prompts, generate the corresponding question prompts.

[0016] In some optional implementations, generating the corresponding question prompt based on the format prompt includes:

[0017] The standard image and the formatted prompt words are input into the third visual language model to generate example prompt information that conforms to the formatted template; the value in the example prompt information is the standard answer determined by the third visual language model.

[0018] By combining the formatted prompts and the example prompts, a prompt for the corresponding question is generated.

[0019] In some optional implementations, the step of inputting the perceptual prediction question prompt and a preset sample image into a first visual language model to generate perceptual prediction information for the sample image includes:

[0020] The sample image and the first question prompt words corresponding to the perception questions in the perception question set are input into the first visual language model to determine the first answer information corresponding to each perception question;

[0021] The sample image and the second question prompt words corresponding to the predicted questions in the predicted question set are input into the first visual language model to determine the second answer information corresponding to each predicted question;

[0022] By integrating the first answer information and the second answer information, perceptual prediction information of the sample image is obtained.

[0023] In some optional implementations, generating descriptive text for the sample image based on the perceptual prediction information and the decision information includes:

[0024] By integrating the perceived prediction information and the decision information, the initial text of the sample image is obtained;

[0025] The initial text is validated and optimized to generate descriptive text that conforms to the target scenario.

[0026] In some optional implementations, the step of performing validation and optimization processing on the initial text to generate descriptive text that conforms to the target scenario includes:

[0027] The initial text is formatted and corrected to generate corrected text;

[0028] The corrected text is broken down into multiple subtexts; each subtext includes at least one sentence.

[0029] Each sub-text is determined to be consistent with the sample image, and a descriptive text that matches the target scene is generated based on the sub-text that matches the sample image.

[0030] In some optional implementations, the step of determining whether each of the sub-texts matches the sample image, and generating descriptive text that conforms to the target scene based on the sub-texts that match the sample image, includes:

[0031] The degree of fit between each sub-text and the sample image is determined according to the fourth visual language model;

[0032] If the fit is greater than a preset threshold, the sub-text is determined to fit the sample image.

[0033] If the number of sub-texts in the correction text that do not match the sample image exceeds a preset number, the correction text is deleted.

[0034] If the number of sub-texts that do not match the sample image in the corrected text does not exceed a preset number, a descriptive text that matches the target scene is generated based on the sub-texts that match the sample image.

[0035] In a second aspect, the present invention provides an apparatus for constructing image-text pairs, comprising:

[0036] The acquisition module is used to acquire a set of questions in the target scenario; the set of questions includes a set of perception questions, a set of prediction questions, and a set of decision questions.

[0037] The prompt word module is used to construct prompt words for the question set; the prompt words include a first prompt word for the perception question set, a second prompt word for the prediction question set, and a third prompt word for the decision question set.

[0038] A generation module is used to input perceptual prediction question prompts and preset sample images into a first visual language model to generate perceptual prediction information of the sample images; the perceptual prediction question prompts include the first question prompt and / or the second question prompt; using the perceptual prediction information as context, the third question prompt and the sample images are input into a second visual language model to generate decision information of the sample images;

[0039] A construction module is used to generate descriptive text for the sample image based on the perception prediction information and the decision information, and to construct an image-text pair containing the sample image and the descriptive text.

[0040] Thirdly, the present invention provides a computer device, comprising: a memory and a processor, the memory and the processor being communicatively connected to each other, the memory storing computer instructions, and the processor executing the computer instructions to perform the image-text pair construction method described in the first aspect or any corresponding embodiment thereof.

[0041] Fourthly, the present invention provides a computer-readable storage medium storing computer instructions for causing a computer to perform the image-text pair construction method described in the first aspect or any corresponding embodiment thereof.

[0042] Fifthly, the present invention provides a computer program product, including computer instructions for causing a computer to execute the method for constructing image-text pairs as described in the first aspect or any corresponding embodiment thereof.

[0043] This invention constructs a set of questions encompassing perception, prediction, and decision-making, and sets corresponding prompts for each. It first generates perceptual and predictive information using an existing visual language model, then uses this perceptual and predictive information as context to generate decision-making information. Finally, it combines the perceptual and predictive information and the decision-making information to generate descriptive text, forming corresponding image-text pairs. This top-down prompt design for perception, prediction, and decision-making allows for the generation of detailed descriptive text for complex scenes, suitable for customized descriptions of target scenarios. Separate recognition and reasoning for perception, prediction, and decision-making avoids excessive questions that could negatively impact the reasoning performance of the visual recognition model, ensuring more accurate descriptive text and improving the quality of image-text pairs. Attached Figure Description

[0044] To more clearly illustrate the technical solutions in the specific embodiments or related technologies of the present invention, the drawings used in the description of the specific embodiments or related technologies will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0045] Figure 1 This is a flowchart illustrating a method for constructing image-text pairs according to an embodiment of the present invention;

[0046] Figure 2 This is a flowchart illustrating another method for constructing image-text pairs according to an embodiment of the present invention;

[0047] Figure 3 This is a schematic diagram illustrating the process of generating a problem set according to an embodiment of the present invention;

[0048] Figure 4 This is a schematic diagram of constructing question prompts according to an embodiment of the present invention;

[0049] Figure 5 This is a schematic diagram illustrating a process for generating a scene description according to an embodiment of the present invention;

[0050] Figure 6 This is a schematic diagram illustrating a process for validating and optimizing initial text according to an embodiment of the present invention;

[0051] Figure 7 This is a structural block diagram of an image-text pair construction apparatus according to an embodiment of the present invention;

[0052] Figure 8 This is a schematic diagram of the hardware structure of a computer device according to an embodiment of the present invention. Detailed Implementation

[0053] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0054] Currently, image-text pair construction methods generally involve manual annotation or synthesis using other Visual Models (VLMs). Manual annotation is time-consuming and labor-intensive, and different people have inconsistent subjective judgments on the same image, resulting in poor consistency of the image-text pair dataset.

[0055] Using existing VLM (Visual Language Model) to synthesize image-text pairs to train new VLMs can reduce the need for manual annotation and improve training efficiency. However, autonomous driving scenarios are complex, involving intricate traffic conditions and road topologies. Directly generating image-text pairs using VLMs results in poor quality and is unsuitable for visual language models in complex scenarios like autonomous driving.

[0056] This invention provides a method for constructing image-text pairs. It constructs a set of questions encompassing perception, prediction, and decision-making, and sets corresponding prompts for each. Using an existing visual language model, it first generates perceptual and predictive information. Then, using this perceptual and predictive information as context, it generates decision-making information. Finally, it combines the perceptual and predictive information and the decision-making information to generate descriptive text, forming corresponding image-text pairs. This top-down prompt design, involving perception, prediction, and decision-making, allows for the generation of descriptive text with rich details and ensures greater accuracy, thus improving the quality of the image-text pairs.

[0057] According to an embodiment of the present invention, an embodiment of a method for constructing image-text pairs is provided. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Furthermore, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0058] This embodiment provides a method for constructing image-text pairs, which can be applied to computers, servers, etc., such as servers used to train visual language models required for autonomous driving scenarios. Figure 1 This is a flowchart of a method for constructing image-text pairs according to an embodiment of the present invention, such as... Figure 1 As shown, the process includes the following steps.

[0059] Step S101: Obtain the problem set in the target scenario; the problem set includes the perception problem set, the prediction problem set, and the decision problem set.

[0060] Prompts determine the quality of text generated by a visual language model. Therefore, to guide a visual language model to generate high-quality scene description text, it is necessary to design accurate and comprehensive prompts based on the required scene. However, traditional prompts are relatively simple, and for complex target scenes such as autonomous driving, simple prompts cannot cover complex situations. For example, based on traditional prompts such as "describe the objects in the image in as much detail as possible," it is impossible to cover complex traffic scenes, road topology, and descriptions of traffic participants, ultimately resulting in low-quality image-text pairs generated based on VLM.

[0061] In this embodiment, for the target scenario requiring the construction of a text-image pair corpus, a series of questions are set within the target scenario, forming a question set containing multiple questions. Furthermore, the questions are divided according to three aspects: perception, prediction, and planning, generating their respective corresponding question sets, namely, the perception question set, the prediction question set, and the planning question set.

[0062] Among them, questions can be set based on the aspects of interest in perception, prediction, and decision-making in the target scenario.

[0063] For example, taking autonomous driving scenarios as an example, perception-related issues can include: weather, time (day / night), road structure (highways, tunnels, intersections, etc.), traffic topology, traffic signs, traffic lights, roadblocks, and the status of other road users. Furthermore, a fine-grained set of perception questions needs to be designed for each object (such as pedestrians, vehicles, and traffic signs), including attributes such as object type, location, and size. Prediction-related issues can include: the behavioral intentions of other vehicles or pedestrians, and the changing status of traffic lights. Decision-making requires generating a description of the vehicle's next action; the corresponding decision-making question could be: based on current perception and prediction, how should the vehicle proceed?

[0064] The target scenario is a scenario that requires decision-making and control based on the collected environmental information, such as autonomous driving scenario, robot control scenario, embodied intelligence scenario, etc. This embodiment does not limit this.

[0065] Step S102: Construct question prompts corresponding to the question set; the question prompts include the first question prompts corresponding to the perception question set, the second question prompts corresponding to the prediction question set, and the third question prompts corresponding to the decision question set.

[0066] In this embodiment, corresponding prompts are constructed for different sets of questions. Specifically, prompts related to perception questions can be constructed based on the perception question set, i.e., first prompts; similarly, second prompts related to prediction questions can be constructed based on the prediction question set, and third prompts related to decision questions can be constructed based on the decision question set.

[0067] Since each question set contains multiple questions, question prompts can be constructed for the entire question set or for each question individually. To improve the quality of the generated text-image pairs, it is preferable to construct separate question prompts for each question, which will be explained later.

[0068] Step S103: Input the perceptual prediction question prompt and the preset sample image into the first visual language model to generate perceptual prediction information of the sample image; the perceptual prediction question prompt includes a first question prompt and / or a second question prompt.

[0069] When constructing image-text pairs based on a visual language model, it is necessary to generate the text corresponding to the existing image based on the existing visual language model. However, in complex target scenes, if the prompts are relatively simple, the corresponding scene text cannot be generated completely and comprehensively; if the prompts are relatively complex, the visual language model will have to make multiple inference decisions at once, affecting the accuracy of the inference results.

[0070] Therefore, in this embodiment, according to the top-down perception, prediction, and decision-making process, the visual language model is used to generate partial text corresponding to the existing image, and then the overall question is obtained by combining them; wherein, the existing image is called the sample image, and the generated overall text is called the description text (Caption).

[0071] In this embodiment, perception and prediction reasoning are performed first, that is, the answers corresponding to the perception question and the prediction question are determined. Specifically, perception prediction question prompts can be generated based on a first question prompt and / or a second question prompt. For a sample image, the perception prediction question prompt is used as the prompt for this recognition, and together with the sample image, it is input into an existing first visual language model (which can be abbreviated as the first VLM). Based on the first VLM, the sample image is recognized to determine the answers corresponding to the perception question and / or prediction question, thereby generating answer information related to perception prediction, that is, perception prediction information.

[0072] For example, if the perceptual prediction question prompt is "identify traffic lights," and the traffic light in the sample image is red, then based on the first Visual Learning Model (VLM), identifying the sample image can generate the answer that the current traffic light is red. This answer can then be used as the perceptual prediction information. It's important to understand that this example is for illustrative purposes only; actual perceptual prediction information includes much more.

[0073] Step S104: Using the perceived prediction information as context, input the third question prompt and sample image into the second visual language model to generate decision information for the sample image.

[0074] In this embodiment, perceptual prediction information is generated independently based on the first Visual Language Model (VLM). This perceptual prediction information is then used as the context of the sample image, combined with a third question prompt corresponding to the decision question, to identify the decision information corresponding to the sample image. Specifically, the perceptual prediction information, the third question prompt, and the sample image are input into the second visual language model, i.e., the second VLM. Based on the second VLM, the answer to the corresponding decision question can be generated, and this answer can serve as the corresponding decision information.

[0075] In the process of generating decision information, the perceived prediction information serves as the known context. Therefore, the process does not require recognition reasoning related to perception decision. At this time, the second VLM only needs to focus on the reasoning related to decision, which can ensure the reasoning effect of the second VLM and make the generated decision information more accurate.

[0076] It is understood that the second VLM is also an existing VLM. The first VLM and the second VLM can be the same VLM or different VLMs. This embodiment does not limit this.

[0077] Step S105: Generate descriptive text for the sample image based on the perception prediction information and decision information, and construct an image-text pair containing the sample image and the descriptive text.

[0078] In this embodiment, the perception and prediction information includes the perception text and prediction text corresponding to the sample image, while the decision information includes the decision text corresponding to the sample image. Therefore, by combining the perception and prediction information and the decision information, text related to perception, prediction, and decision-making can be generated, which can then serve as the descriptive text for the sample image. For example, the text content in the perception and prediction information and the decision information can be merged into the descriptive text.

[0079] In summary, for a given sample image, its corresponding descriptive text can be generated based on the above process, thereby constructing a corresponding image-text pair. This image-text pair represents the correspondence between the sample image and the descriptive text. Furthermore, this image-text pair can be used to construct a corpus, which can be used to train a Visual Model for the target scene, thus obtaining a Visual Model suitable for the target scene.

[0080] The image-text pair construction method provided in this embodiment constructs a set of questions covering perception, prediction, and decision-making, and sets corresponding prompts for each. It first generates perception and prediction information using an existing visual language model, then uses this perception and prediction information as context to generate decision information. Finally, it combines the perception and prediction information and the decision information to generate descriptive text, forming corresponding image-text pairs. Through top-down prompt design of perception, prediction, and decision-making, it can generate descriptive text containing rich details in complex scenes, suitable for customized descriptions of target scenes. Perception, prediction, and decision-making are performed separately for recognition and reasoning, avoiding excessive questions that could affect the reasoning performance of the visual recognition model, ensuring more accurate generated descriptive text and improving the quality of image-text pairs.

[0081] This embodiment provides another method for constructing image-text pairs, which can be applied to computers, servers, etc., such as servers used to train visual language models required for autonomous driving scenarios. Figure 2This is a flowchart of a method for constructing image-text pairs according to an embodiment of the present invention, such as... Figure 2 As shown, the process includes the following steps.

[0082] Step S201: Obtain the problem set in the target scenario; the problem set includes the perception problem set, the prediction problem set, and the decision problem set.

[0083] Please see details Figure 1 Step S101 of the illustrated embodiment will not be described again here.

[0084] In some alternative implementations, since there are many problems in the target scenario, in order to improve the efficiency of generating a problem set, a simple problem set can be initially generated first, and then the problem set can be expanded to generate a more complete problem set.

[0085] Specifically, step S201, "obtaining the problem set in the target scenario," may include: obtaining a first problem set related to perception problems, a second problem set related to prediction problems, and a third problem set related to decision-making problems in the target scenario. The first, second, and third problem sets are then expanded to generate corresponding perception problem sets, prediction problem sets, and decision problem sets.

[0086] In this embodiment, as Figure 3 As shown, an initial question set for the target scenario can be generated through question collection or manual specification. This question set includes a first question set related to perception, a second question set related to prediction, and a third question set related to decision-making. For example, for an autonomous driving scenario, key elements such as weather, traffic signals, signs, pedestrians, vehicles, and road structures can be identified, and an initial question set can be generated based on these elements.

[0087] After obtaining the initial problem set, the problems in the problem set are expanded. For example, each problem can be expanded to add details and contextual information to make it more comprehensive; multiple variations can be generated for each problem to increase its diversity; and the language of the problems can be polished and the logic strengthened to improve the quality of the problem set. For example, the initial problem set can be expanded based on a Large Language Model (LLM).

[0088] The expanded question set contains more detailed questions in the target scenario, and this question set can be used as the final question set. Accordingly, the first expanded question set is the perception question set, the second expanded question set is the prediction question set, and the third expanded question set is the decision question set.

[0089] Step S202: Construct question prompts corresponding to the question set; the question prompts include the first question prompts corresponding to the perception question set, the second question prompts corresponding to the prediction question set, and the third question prompts corresponding to the decision question set.

[0090] Specifically, step S202, “constructing the question prompt words corresponding to the question set”, may include steps S2021 to S2023.

[0091] S2021. For each question in the question set, construct a format template corresponding to the question. The format template is a key-value pair template, and the key in the format template includes the key information of the question, while the value in the format template is used to represent the answer to the question.

[0092] When generating text from sample images based on visual language models, the "illusion problem" may occur. Illusion refers to a discrepancy between the content generated by the model and verifiable real-world facts, or between the content generated by the model and the user's instructions or context. To reduce the occurrence of illusions, a unified format prompt instruction, i.e., a format template, is designed.

[0093] In this embodiment, the problem set contains multiple problems, and a corresponding format template can be constructed for each problem. For example, a corresponding format template can be constructed for each perception problem in the perception problem set, a corresponding format template can be constructed for each prediction problem in the prediction problem set, and a corresponding format template can also be constructed for each decision problem in the decision problem set.

[0094] Furthermore, the format template is a key-value pair template, where the key in the format template includes key information about the question, and the value in the format template represents the answer to the question.

[0095] Specifically, for questions within a question set, key information can be extracted. For example, the question itself can be used as key information, or keywords can be extracted from the question to generate key information. The value in the format template primarily indicates the format used by the visual language model when outputting. The value in the format template can include all candidate answers or be empty, depending on the specific question.

[0096] The template can be in JSON format, with the key in the JSON format customized based on the question, and the value can be generated by the Visual Language Model (VLM) autoregressive.

[0097] S2022, combine the question and the corresponding format template to generate format prompts; the format prompts indicate that the answer to the question should be generated according to the corresponding format template.

[0098] In this embodiment, for each question in the question set, a corresponding prompt word, i.e., a format prompt word, can be generated by combining the question with the corresponding format template. Furthermore, the meaning of this format prompt word is: generate the answer to the question according to the corresponding format template; that is, if the visual language model is instructed to generate information based on this format prompt word, then when the visual language model generates the answer, it needs to generate key-value pair information according to this format template.

[0099] For example, if the question is "How does the traffic light ahead change?", the key information is the traffic light itself. A JSON format template can be designed for this question: {"traffic light":[red / greeen / yellow]}.

[0100] Based on this, the following formatted prompt words can be generated:

[0101] Please output the changes in the traffic lights in the following JSON format:

[0102] {

[0103] "Traffic Lights": [red / greeen / yellow]

[0104] }

[0105] If the visual language model recognizes the sample image based on the formatted prompt, and the traffic light in the sample image is red, then the output of the visual language model can be: {"traffic light":"red"}.

[0106] S2023, generate the corresponding question prompt based on the format prompt.

[0107] In this embodiment, for a given question's formatted prompt, that prompt can be directly used as the prompt for subsequent use, i.e., the question prompt. Alternatively, the formatted prompt can be improved to generate a more effective question prompt.

[0108] Optionally, step S2023 above, "generating the corresponding question prompt based on the format prompt," may include steps A1 to A2.

[0109] Step A1: Input the standard image and formatted prompts into the third visual language model to generate example prompts that conform to the formatted template; the value in the example prompts is the standard answer determined by the third visual language model.

[0110] Step A2: Combine the format prompts and example prompts to generate the corresponding question prompts.

[0111] In this embodiment, to further reduce the occurrence of hallucinations and ensure the accuracy of the content generated by the VLM, some example prompts, i.e. example prompt information, are also established. Combining format prompt words and example prompt information, question prompt words are generated. When the VLM generates inference results based on the question prompt words, it can refer to the example prompt information, thereby guiding the visual language model to generate high-quality scene descriptions and obtain customized descriptions for autonomous driving scenarios, which can ensure the accuracy and naturalness of the generated text.

[0112] Specifically, high-quality images can be selected from existing image databases and used as representative standard images. Then, using an existing visual language model, namely the third visual language model (abbreviated as Third VLM), the answer to the question is generated according to the formatted prompts corresponding to the question. This answer is represented as key-value pairs of a formatted template. In the key-value pairs output by Third VLM, the value is the answer to the corresponding question. This answer can serve as a standard answer, thus obtaining example prompt information that conforms to the formatted template. It can be understood that this example prompt information is in key-value pair format, where the key is the key information of the question, and the value is the standard answer generated by Third VLM.

[0113] For answers directly generated by the third VLM, these answers can be optimized and used as the standard answers. Since only a small amount of example hints and a limited number of standard images are needed, the answers can be polished through manual review and then used as the standard answers in various scenarios to generate example hints. These example hints are then combined to generate question prompts, allowing them to serve as supplementary background knowledge for the visual language model, thereby guiding the model to output correct, detailed, and consistent descriptions.

[0114] Since the example prompts are mainly used as background knowledge and are also a template, the same example prompts can be used for different problem prompts such as perception, prediction, and decision-making; or, example prompts applicable to different scenarios such as perception, prediction, and decision-making can be generated separately, and then corresponding problem prompts can be generated. This embodiment does not limit this.

[0115] Figure 4 This illustrates one method for constructing question prompts. For example... Figure 4 As shown, the final question set is determined, and a JSON format template is set for each question. Initial examples are generated using high-quality standard images and a third-party Visual Model (VLM). The answers to these examples are then refined to produce sample hints in JSON format. Combining the question set, format templates, and sample hints, question hints for each question can be constructed.

[0116] It is understood that the third VLM can be the same VLM as the first VLM and the second VLM mentioned above, or it can be a different VLM, depending on the actual situation.

[0117] Step S203: Input the perceptual prediction question prompt and the preset sample image into the first visual language model to generate perceptual prediction information of the sample image; the perceptual prediction question prompt includes a first question prompt and / or a second question prompt.

[0118] Please see details Figure 1 Step S103 of the illustrated embodiment will not be described again here.

[0119] In some optional implementations, the above step S203, "inputting the perceptual prediction question prompt and the preset sample image into the first visual language model to generate perceptual prediction information of the sample image", may include steps B1 to B3.

[0120] Step B1: Input the sample images and the first question prompts corresponding to the perception questions in the perception question set into the first visual language model to determine the first answer information corresponding to each perception question.

[0121] Step B2: Input the sample images and the second question prompts corresponding to the predicted questions in the prediction question set into the first visual language model to determine the second answer information corresponding to each predicted question.

[0122] Step B3: Integrate the first answer information and the second answer information to obtain the perceptual prediction information of the sample image.

[0123] In this embodiment, both the perception question set and the prediction question set contain a large number of questions. The first VLM is used to identify each question separately, so that the first VLM can focus its attention on a single question and generate the corresponding answer to the question more accurately, thereby realizing the perception and prediction of the sample image.

[0124] Specifically, if the perception problem set contains m perception problems, corresponding first question prompts can be set for each of these m perception problems, resulting in a total of m first question prompts. For example, each first question prompt can be constructed based on steps S2021 to S2023 above. The sample image to be identified and a first question prompt are input into a first VLM. This first VLM can focus only on a single perception problem, thereby accurately determining the answer to that perception problem and generating the corresponding first answer information. For other perception problems, the same method can be used to determine the corresponding first answer information.

[0125] Similarly, if the prediction question set contains n prediction questions, corresponding second question prompts can be set for each of these n prediction questions, resulting in a total of n second question prompts. For example, each second question prompt can be constructed based on steps S2021 to S2023 above. The sample image to be identified and a second question prompt are input into the second VLM, which can focus only on the single prediction question, thereby accurately determining the answer to that prediction question and generating the corresponding second answer information. For other prediction questions, the same method can be used to determine the corresponding second answer information.

[0126] Through the above processing, the first answer information corresponding to m perception questions and the second answer information corresponding to n prediction questions can be determined. By integrating these first answer information and second answer information, the perception prediction information of the sample image can be obtained.

[0127] The first answer information and the second answer information can be information in the form of key-value pairs, or information in the form of text converted from key-value pairs. This embodiment does not limit this.

[0128] Optionally, since some prediction results are related to the corresponding perceived content, in step B2 above, the first answer information related to perception can be used as background knowledge, and the sample image and the second question prompt words corresponding to the predicted questions in the prediction question set can be input into the first VLM, so that the second answer information corresponding to each predicted question can be determined more accurately by combining the recognized perceived information (i.e., the first answer information).

[0129] Step S204: Using the perceived prediction information as context, input the third question prompt and sample image into the second visual language model to generate decision information for the sample image.

[0130] Please see details Figure 1 Step S104 of the illustrated embodiment will not be described again here.

[0131] Each decision question also corresponds to a third question prompt. Similar to steps B1 and B2 above, corresponding answer information can be determined for each decision question, and finally, the decision information is generated by combining the results. This decision information can represent the corresponding decision action.

[0132] Step S205: Generate descriptive text for the sample image based on the perception prediction information and decision information, and construct an image-text pair containing the sample image and the descriptive text.

[0133] Please see details Figure 1 Step S105 of the illustrated embodiment will not be described again here.

[0134] Optionally, step S205, "generating descriptive text of sample images based on perception prediction information and decision information," may include steps C1 to C2.

[0135] Step C1: Integrate the perceptual prediction information and decision information to obtain the initial text of the sample image.

[0136] Step C2 involves validating and optimizing the initial text to generate descriptive text that matches the target scenario.

[0137] In this embodiment, the perceptual prediction information and decision information generated by VLM are integrated to obtain preliminary text that can describe the sample image, i.e., initial text. Since the perceptual prediction information and decision information generated by VLM are in key-value pair form (or information extracted from key-value pairs), this information can be optimized to generate the initial text.

[0138] Figure 5 This illustrates a process for generating scene descriptions, such as... Figure 5 As shown, the perceptual prediction information and decision information generated by the first VLM and the second VLM can be optimized based on the Large Language Model (LLM) to improve the fluency and clarity of the language and generate an initial version of the scene description of the sample image, i.e., the initial text.

[0139] Since the various pieces of information in the initial text are generated separately, the generation effect of each aspect—perception, prediction, and decision-making—can be preliminarily judged during the initial text generation process, and information with poor performance can be deleted, making the operation quite flexible. For example, if the decision-making effect of the visual recognition model is poor, the corresponding decision-making information can be deleted during the initial text generation, and the initial text of the sample image can be generated only based on the perception and prediction information.

[0140] The initial text can be further validated and optimized based on the requirements of the target scene to generate text suitable for the target scene. This text can then be used to describe the sample image, i.e., the descriptive text.

[0141] Optionally, step C2, "to perform validation and optimization on the initial text and generate descriptive text that conforms to the target scenario," may include steps C21 to C23.

[0142] Step C21: Correct the format of the initial text to generate corrected text.

[0143] Step C22 involves breaking down the corrected text into multiple subtexts; each subtext includes at least one sentence.

[0144] Step C23: Determine whether each sub-text matches the sample image, and generate descriptive text that matches the target scene based on the sub-text that matches the sample image.

[0145] In this embodiment, in order to ensure the quality and consistency of the generated text-image pairs, it is necessary to perform quality checks on the generated content. Figure 6 This diagram illustrates a process for validating and optimizing initial text. Figure 6 As shown, a Large Language Model (LLM) can be used to check whether the generated initial text meets the preset format and content requirements, and optimize the content that does not meet the format requirements to obtain the format-corrected text, i.e., the corrected text.

[0146] Because there are many issues related to perception, prediction, and decision-making in the target scenario, the generated initial and corrected texts are both large blocks of text. Directly validating these large blocks of corrected text results in inaccurate results. For example, if a Visual Modeling Model (VLM) is used to quality inspect the entire text paragraph, the model's performance will not be good. Therefore, in this embodiment, the entire corrected text is decomposed into multiple individual sub-texts, each containing one or more sentences. For example, a sub-text may contain only one sentence. Subsequently, each sub-text is validated based on the VLM, allowing the model to focus more on individual sentences, which is more accurate than quality inspecting the entire paragraph.

[0147] Specifically, such as Figure 6 As shown, a Large Language Model (LLM) can be used to decompose the corrected text into multiple sentences with distinct semantics, forming multiple sub-texts. Then, each sub-text is evaluated to determine if it matches the sample image, thus more accurately judging the suitability of the generated sub-texts. Finally, the sub-text that matches the sample image is selected, and the final version of the text is generated, which can then serve as the descriptive text for the target scene.

[0148] Optionally, step C23, "determine whether each sub-text matches the sample image, and generate descriptive text that matches the target scene based on the sub-text that matches the sample image," may include steps C231 to C234.

[0149] Step C231: Determine the fit between each sub-text and the sample image based on the fourth visual language model.

[0150] Step C232: If the fit is greater than a preset threshold, determine that the subtext matches the sample image.

[0151] Step C233: If the number of sub-texts in the correction text that do not match the sample image exceeds a preset number, delete the correction text.

[0152] Step C234: If the number of sub-texts in the corrected text that do not match the sample image does not exceed a preset number, generate descriptive text that matches the target scene based on the sub-texts that match the sample image.

[0153] In this embodiment, when performing quality inspection on a single sub-text, such as Figure 6 As shown, the fourth visual language model (i.e., the fourth VLM) is used to check the fit between each subtext and its corresponding sample image, and a preset threshold is set for the fit. For example, the fit range is 0 to 100, and the preset threshold can be 90, 95, etc. If the fit between the subtext and the sample image is greater than the preset threshold, they are considered to be a good match; otherwise, they are not a good match.

[0154] Furthermore, a preset number is set. If the number of sub-texts that do not match the sample image exceeds this preset number, it indicates that there are too many mismatched sub-texts in the currently generated corrected text, and the overall quality of the corrected text is poor. Therefore, the corrected text can be directly deleted, i.e., no corresponding image-text pair is generated. Conversely, if the number of sub-texts that do not match the sample image does not exceed the preset number, it can be considered that these sub-texts as a whole meet the requirements. In this case, the sub-texts with poor matching degree are removed, and descriptive text that matches the target scene is generated based on the sub-texts that match the sample image.

[0155] The preset number can be a fixed value or a certain proportion. The appropriate preset number is determined based on the total number of sub-texts decomposed from the corrected text. The size of the preset number is related to the quality requirements of the image-text pair. For example, for image-text pairs with high requirements, the preset number can be set to 1, that is, if there are mismatched sub-texts, the image-text pair corresponding to the sample image will be removed.

[0156] In this embodiment, by decomposing the text into multiple sub-texts and performing quality checks on each sub-text separately, the accuracy of the quality checks can be ensured. Using VLM to validate each sub-text allows for the selection of sub-texts with high relevance, and the direct deletion of sub-texts when there are a large number of non-relevant ones. This eliminates poor-quality image-text pairs, ensuring both the quality and consistency of the generated corpus and making the final descriptive text as consistent as possible with the actual target scenario.

[0157] It is understood that the fourth VLM can be the same VLM as the first, second, and third VLMs mentioned above, or it can be a different VLM, depending on the specific circumstances. Furthermore, the LLM used for formatting correction, text decomposition, and generating the final text can be the same LLM or different LLMs. For example, the same LLM can be used, with different prompts used to set corresponding tasks for the LLM, thereby achieving functions such as formatting correction and text decomposition.

[0158] The image-text pair construction method provided in this embodiment designs key-value pair format templates for each question, ensuring the consistency and readability of scene descriptions generated by visual language models (e.g., the first VLM and the second VLM), facilitating subsequent processing and analysis, and reducing the occurrence of visual illusions. Combining example prompts with generated question prompt words provides richer background knowledge for the VLM, effectively guaranteeing the quality of the VLM's output text. Based on these question prompt words, the visual language model can be guided to generate high-quality scene descriptions, obtaining customized descriptions specific to scenarios such as autonomous driving, while ensuring the accuracy and naturalness of the generated text. Furthermore, this method has excellent scalability and adaptability, enabling the construction of high-quality corpora for various scenarios, and subsequently training visual language models for the corresponding scenarios based on these corpora.

[0159] This embodiment also provides an image-text pair construction apparatus for implementing the above embodiments and preferred embodiments, and details already described will not be repeated. As used below, the term "module" can be a combination of software and / or hardware that implements a predetermined function. Although the apparatus described in the following embodiments is preferably implemented in software, hardware implementation, or a combination of software and hardware, is also possible and contemplated.

[0160] This embodiment provides an apparatus for constructing image-text pairs, such as... Figure 7 As shown, it includes:

[0161] The acquisition module 701 is used to acquire a set of questions in the target scenario; the set of questions includes a set of perception questions, a set of prediction questions, and a set of decision questions.

[0162] The prompt word module 702 is used to construct prompt words corresponding to the question set; the prompt words include a first prompt word corresponding to the perception question set, a second prompt word corresponding to the prediction question set, and a third prompt word corresponding to the decision question set;

[0163] The generation module 703 is used to input the perceptual prediction question prompts and the preset sample image into the first visual language model to generate perceptual prediction information of the sample image; the perceptual prediction question prompts include the first question prompt and / or the second question prompt; using the perceptual prediction information as context, the third question prompt and the sample image are input into the second visual language model to generate decision information of the sample image;

[0164] The construction module 704 is used to generate descriptive text for the sample image based on the perception prediction information and the decision information, and to construct an image-text pair containing the sample image and the descriptive text.

[0165] In some optional implementations, the prompt word module 702 constructs the prompt words corresponding to the question set, including:

[0166] For each question in the question set, a format template corresponding to the question is constructed; the format template is a key-value pair template, and the key in the format template includes the key information of the question, and the value in the format template is used to represent the answer to the question;

[0167] Based on the question and the corresponding format template, generate format prompts; the format prompts indicate that the answer to the question should be generated according to the corresponding format template.

[0168] Based on the specified format prompts, generate the corresponding question prompts.

[0169] In some optional implementations, the prompt word module 702 generates a question prompt word for the corresponding question based on the format prompt word, including:

[0170] The standard image and the formatted prompt words are input into the third visual language model to generate example prompt information that conforms to the formatted template; the value in the example prompt information is the standard answer determined by the third visual language model.

[0171] By combining the formatted prompts and the example prompts, a prompt for the corresponding question is generated.

[0172] In some optional implementations, the generation module 703 inputs the perceptual prediction question prompt and a preset sample image into the first visual language model to generate perceptual prediction information for the sample image, including:

[0173] The sample image and the first question prompt words corresponding to the perception questions in the perception question set are input into the first visual language model to determine the first answer information corresponding to each perception question;

[0174] The sample image and the second question prompt words corresponding to the predicted questions in the predicted question set are input into the first visual language model to determine the second answer information corresponding to each predicted question;

[0175] By integrating the first answer information and the second answer information, perceptual prediction information of the sample image is obtained.

[0176] In some optional implementations, the construction module 704 generates descriptive text for the sample image based on the perceptual prediction information and the decision information, including:

[0177] By integrating the perceived prediction information and the decision information, the initial text of the sample image is obtained;

[0178] The initial text is validated and optimized to generate descriptive text that conforms to the target scenario.

[0179] In some optional implementations, the construction module 704 performs validation and optimization processing on the initial text to generate descriptive text that conforms to the target scenario, including:

[0180] The initial text is formatted and corrected to generate corrected text;

[0181] The corrected text is broken down into multiple subtexts; each subtext includes at least one sentence.

[0182] Each sub-text is determined to be consistent with the sample image, and a descriptive text that matches the target scene is generated based on the sub-text that matches the sample image.

[0183] In some optional implementations, the construction module 704 determines whether each of the sub-texts matches the sample image, and generates descriptive text that conforms to the target scene based on the sub-texts that match the sample image, including:

[0184] The degree of fit between each sub-text and the sample image is determined according to the fourth visual language model;

[0185] If the fit is greater than a preset threshold, the sub-text is determined to fit the sample image.

[0186] If the number of sub-texts in the correction text that do not match the sample image exceeds a preset number, the correction text is deleted.

[0187] If the number of sub-texts that do not match the sample image in the corrected text does not exceed a preset number, a descriptive text that matches the target scene is generated based on the sub-texts that match the sample image.

[0188] Further functional descriptions of the above modules and units are the same as those in the corresponding embodiments described above, and will not be repeated here.

[0189] In this embodiment, the image-text pair construction device is presented in the form of a functional unit. Here, a unit refers to an ASIC (Application Specific Integrated Circuit) circuit, including a processor and memory that execute one or more software or fixed programs, and / or other devices that can provide the above functions.

[0190] This invention also provides a computer device having the above-described features. Figure 7 The apparatus for constructing image-text pairs is shown.

[0191] Please see Figure 8 , Figure 8 This is a schematic diagram of the structure of a computer device provided in an optional embodiment of the present invention, such as... Figure 8 As shown, the computer device includes one or more processors 10, memory 20, and interfaces for connecting the components, including high-speed interfaces and low-speed interfaces. The components communicate with each other via different buses and can be mounted on a common motherboard or otherwise installed as needed. The processors can process instructions executed within the computer device, including instructions stored in or on memory to display graphical information of a GUI on external input / output devices (such as display devices coupled to the interfaces). In some alternative implementations, multiple processors and / or multiple buses can be used with multiple memories, if desired. Similarly, multiple computer devices can be connected, each providing some of the necessary operations (e.g., as a server array, a group of blade servers, or a multiprocessor system). Figure 8 Take a processor 10 as an example.

[0192] Processor 10 may be a central processing unit, a network processor, or a combination thereof. Processor 10 may further include a hardware chip. The hardware chip may be an application-specific integrated circuit (ASIC), a programmable logic device (PLD), or a combination thereof. The programmable logic device may be a complex programmable logic device (CAMP), a field-programmable gate array (FPGA), a general-purpose array logic (GDA), or any combination thereof.

[0193] The memory 20 stores instructions executable by at least one processor 10 to cause the at least one processor 10 to perform the method shown in the above embodiments.

[0194] The memory 20 may include a program storage area and a data storage area. The program storage area may store the operating system and applications required for at least one function; the data storage area may store data created based on the use of the computer device. Furthermore, the memory 20 may include high-speed random access memory and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some alternative embodiments, the memory 20 may optionally include memory remotely located relative to the processor 10, and these remote memories may be connected to the computer device via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0195] The memory 20 may include volatile memory, such as random access memory; the memory may also include non-volatile memory, such as flash memory, hard disk or solid-state drive; the memory 20 may also include a combination of the above types of memory.

[0196] The computer device also includes a communication interface 30 for communicating with other devices or communication networks.

[0197] This invention also provides a computer-readable storage medium. The methods described above according to embodiments of the invention can be implemented in hardware or firmware, or implemented as computer code that can be recorded on a storage medium, or implemented as computer code downloaded via a network and originally stored on a remote storage medium or a non-transitory machine-readable storage medium and then stored on a local storage medium. Thus, the methods described herein can be processed by software stored on a storage medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware. The storage medium can be a magnetic disk, optical disk, read-only memory, random access memory, flash memory, hard disk, or solid-state drive, etc.; further, the storage medium can also include combinations of the above types of memory. It is understood that computers, processors, microprocessor controllers, or programmable hardware include storage components capable of storing or receiving software or computer code, which, when accessed and executed by the computer, processor, or hardware, implements the methods shown in the above embodiments.

[0198] A portion of this invention can be applied as a computer program product, such as computer program instructions, which, when executed by a computer, can invoke or provide the methods and / or technical solutions according to the invention through the operation of the computer. Those skilled in the art will understand that the forms in which computer program instructions exist in a computer-readable medium include, but are not limited to, source files, executable files, installation package files, etc. Correspondingly, the ways in which computer program instructions are executed by a computer include, but are not limited to: the computer directly executing the instructions, or the computer compiling the instructions and then executing the corresponding compiled program, or the computer reading and executing the instructions, or the computer reading and installing the instructions and then executing the corresponding installed program. Here, the computer-readable medium can be any available computer-readable storage medium or communication medium accessible to a computer.

[0199] Although embodiments of the present invention have been described in conjunction with the accompanying drawings, those skilled in the art can make various modifications and variations without departing from the spirit and scope of the present invention, and such modifications and variations should all be covered within the protection scope of the present invention.

Claims

1. A method for constructing image-text pairs, characterized in that, The method includes: Obtain a set of questions in the target scenario; the set of questions includes a set of perception questions, a set of prediction questions, and a set of decision questions. Construct question prompts corresponding to the question set; the question prompts include first question prompts related to perception questions corresponding to the perception question set, second question prompts related to prediction questions corresponding to the prediction question set, and third question prompts related to decision questions corresponding to the decision question set; The perceptual prediction question prompts and preset sample images are input into a first visual language model to generate perceptual prediction information for the sample images; the perceptual prediction question prompts include the first question prompt and / or the second question prompt. Using the perceived prediction information as context, the third question prompt and the sample image are input into the second visual language model to generate decision information for the sample image; Based on the perception prediction information and the decision information, a descriptive text for the sample image is generated, and an image-text pair containing the sample image and the descriptive text is constructed.

2. The method according to claim 1, characterized in that, The construction of the question prompt words corresponding to the question set includes: For each question in the question set, a format template corresponding to the question is constructed; the format template is a key-value pair template, and the key in the format template includes the key information of the question, and the value in the format template is used to represent the answer to the question; Based on the question and the corresponding format template, generate format prompts; the format prompts indicate that the answer to the question should be generated according to the corresponding format template. Based on the specified format prompts, generate prompts for the corresponding questions.

3. The method according to claim 2, characterized in that, The step of generating question prompts for the corresponding questions based on the specified format prompts includes: The standard image and the formatted prompt words are input into the third visual language model to generate example prompt information that conforms to the formatted template; the value in the example prompt information is the standard answer determined by the third visual language model. By combining the formatted prompts and the example prompts, a prompt for the corresponding question is generated.

4. The method according to claim 1, characterized in that, The step of inputting the perceptual prediction question prompts and preset sample images into the first visual language model to generate perceptual prediction information for the sample images includes: The sample image and the first question prompt words corresponding to the perception questions in the perception question set are input into the first visual language model to determine the first answer information corresponding to each perception question; The sample image and the second question prompt words corresponding to the predicted questions in the predicted question set are input into the first visual language model to determine the second answer information corresponding to each predicted question; By integrating the first answer information and the second answer information, perceptual prediction information of the sample image is obtained.

5. The method according to claim 1, characterized in that, The step of generating descriptive text for the sample image based on the perceptual prediction information and the decision information includes: By integrating the perceived prediction information and the decision information, the initial text of the sample image is obtained; The initial text is validated and optimized to generate descriptive text that conforms to the target scenario.

6. The method according to claim 5, characterized in that, The step of performing validation and optimization processing on the initial text to generate descriptive text that conforms to the target scenario includes: The initial text is formatted and corrected to generate corrected text; The corrected text is broken down into multiple subtexts; each subtext includes at least one sentence. Each sub-text is determined to be consistent with the sample image, and a descriptive text that matches the target scene is generated based on the sub-text that matches the sample image.

7. The method according to claim 6, characterized in that, The step of determining whether each of the sub-texts matches the sample image, and generating descriptive text that conforms to the target scene based on the sub-texts that match the sample image, includes: The degree of fit between each sub-text and the sample image is determined according to the fourth visual language model; If the fit is greater than a preset threshold, the sub-text is determined to fit the sample image. If the number of sub-texts in the correction text that do not match the sample image exceeds a preset number, the correction text is deleted. If the number of sub-texts that do not match the sample image in the corrected text does not exceed a preset number, a descriptive text that matches the target scene is generated based on the sub-texts that match the sample image.

8. An apparatus for constructing image-text pairs, characterized in that, The device includes: The acquisition module is used to acquire a set of questions in the target scenario; the set of questions includes a set of perception questions, a set of prediction questions, and a set of decision questions. The prompt word module is used to construct prompt words for the question set; the prompt words include first prompt words related to perception questions corresponding to the perception question set, second prompt words related to prediction questions corresponding to the prediction question set, and third prompt words related to decision questions corresponding to the decision question set. A generation module is used to input perceptual prediction question prompts and preset sample images into a first visual language model to generate perceptual prediction information of the sample images; the perceptual prediction question prompts include the first question prompt and / or the second question prompt; using the perceptual prediction information as context, the third question prompt and the sample images are input into a second visual language model to generate decision information of the sample images; A construction module is used to generate descriptive text for the sample image based on the perception prediction information and the decision information, and to construct an image-text pair containing the sample image and the descriptive text.

9. A computer device, characterized in that, include: A memory and a processor are communicatively connected, the memory storing computer instructions, and the processor executing the computer instructions to perform the method for constructing image-text pairs according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions for causing a computer to perform the method for constructing image-text pairs according to any one of claims 1 to 7.

Images