Data processing method, device, equipment, storage medium and program product

Abstract

The present invention provides a data processing method, device, equipment, storage medium and program product, which are applied to the field of data processing technology. The method includes: obtaining an image to be processed; inputting the image to be processed into a single image-text model to obtain corresponding text description information, inputting the image to be processed into an image segmentation model to obtain object category information in the image to be processed; inputting the text description information and the object category information into a large language model to generate multi-object related question and answer information, the multi-object related question and answer information includes image understanding information and object location information; performing image backfilling on the objects in the multi-object related question and answer information to obtain first image-text data; inputting the image to be processed and the multi-object related question and answer information into a reward model to obtain an inspection item score; determining whether to determine the first image-text data as multimodal training data according to the inspection item score.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a data processing method, device, equipment, storage medium and program product. Background Art

[0002] In order to train a large multimodal language model for multi-image and text scenarios, a large amount of data is required as a training basis. In the process of building training data, it is crucial to ensure data quality.

[0003] In existing technologies, data screening often relies on manual labeling or simple rules. However, these data screening methods are not only inefficient, but also difficult to fully capture complex errors in the data, so the data quality is poor. Summary of the invention

[0004] The present invention provides a data processing method, device, equipment, storage medium and program product to solve the problem that the data screening method in the prior art is not only inefficient but also difficult to fully capture complex errors in the data and the data quality is poor.

[0005] The present invention provides a data processing method, comprising: acquiring an image to be processed; inputting the image to be processed into a single image-text model to obtain corresponding text description information, and inputting the image to be processed into an image segmentation model to obtain object category information in the image to be processed; inputting the text description information and the object category information into a large language model to generate multi-object associated question and answer information, wherein the multi-object associated question and answer information includes image understanding information and object position information; performing image backfilling on objects in the multi-object associated question and answer information to obtain first image-text data; inputting the image to be processed and the multi-object associated question and answer information into a reward model to obtain an inspection item score; and determining whether to determine the first image-text data as multimodal training data according to the inspection item score.

[0006] According to a data processing method provided by the present invention, the reward model is used to score each inspection item and generate an inspection item score based on the scores of all inspection items; the determining whether to determine the first graphic data as multimodal training data based on the inspection item score includes: when the inspection item score is greater than a first threshold, determining the first graphic data as multimodal training data.

[0007] According to a data processing method provided by the present invention, before determining the first graphic data as multimodal training data, the method also includes: inputting the first graphic data into a data refinement module to obtain processed first graphic data; wherein the data refinement module is used to remove erroneous data described in the first graphic data.

[0008] According to a data processing method provided by the present invention, the error categories of the first graphic data include object hallucination, attribute error, object relationship error, object position error and unreasonable image guessing.

[0009] According to a data processing method provided by the present invention, before inputting the image to be processed and the multi-object related question and answer information into a reward model to obtain the inspection item score, the method also includes: obtaining training data, wherein the training data is data obtained after performing sentence-level annotation processing on the multi-object related question and answer information; and fine-tuning the visual language model based on the training data to obtain the reward model.

[0010] According to a data processing method provided by the present invention, the input data of the visual language model includes a first sentence and a sentence to be scored, and the first sentence contains context information of the sentence to be scored.

[0011] The present invention also provides a data processing device, comprising the following modules: an acquisition module and a processing module; the acquisition module is used to acquire an image to be processed; the processing module is used to input the image to be processed into a single image-text model to obtain corresponding text description information, and input the image to be processed into an image segmentation model to obtain object category information in the image to be processed; the text description information and the object category information are input into a large language model to generate multi-object related question and answer information, and the multi-object related question and answer information includes image understanding information and object location information; image backfilling is performed on the objects in the multi-object related question and answer information to obtain first image-text data; the image to be processed and the multi-object related question and answer information are input into a reward model to obtain an inspection item score; and it is determined whether to determine the first image-text data as multimodal training data according to the inspection item score.

[0012] According to a data processing device provided by the present invention, the reward model is used to score each inspection item and generate an inspection item score based on the scores of all inspection items; the processing module is used to determine the first graphic data as multimodal training data when the inspection item score is greater than a first threshold.

[0013] According to a data processing device provided by the present invention, the processing module is used to input the first graphic data into the data refinement module to obtain processed first graphic data; wherein the data refinement module is used to remove erroneous data in the first graphic data.

[0014] According to a data processing device provided by the present invention, the error categories of the first graphic data include object hallucination, attribute error, object relationship error, object position error and unreasonable image guessing.

[0015] According to a data processing device provided by the present invention, before the image to be processed and the multi-object related question and answer information are input into a reward model to obtain the inspection item score, the acquisition module is used to obtain training data, and the training data is data obtained after sentence-level annotation processing of the multi-object related question and answer information; the processing module is used to fine-tune the visual language model based on the training data to obtain the reward model.

[0016] According to a data processing device provided by the present invention, the input data of the visual language model includes a first sentence and a sentence to be scored, and the first sentence contains context information of the sentence to be scored.

[0017] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, any of the above-mentioned data processing methods is implemented.

[0018] The present invention also provides a non-transitory computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the data processing method described above is implemented.

[0019] The present invention also provides a computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the computer program implements any of the above-mentioned data processing methods.

[0020] The data processing method, device, equipment, storage medium and program product provided by the present invention can obtain an image to be processed; input the image to be processed into a single image-text model to obtain corresponding text description information, input the image to be processed into an image segmentation model to obtain object category information in the image to be processed; input the text description information and the object category information into a large language model to generate multi-object related question and answer information, the multi-object related question and answer information includes image understanding information and object position information; perform image backfilling on the objects in the multi-object related question and answer information to obtain first image-text data; input the image to be processed and the multi-object related question and answer information into a reward model to obtain an inspection item score; determine whether to determine the first image-text data as multimodal training data according to the inspection item score. Through this scheme, since the generated first image-text data can be scored and screened through the reward model, it can not only improve the data screening efficiency, but also improve the description accuracy of the multimodal training data, thereby improving the quality of the training data, and providing a solid data foundation for training a multimodal large language model for multi-image-text scenes. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0022] Figure 1 It is one of the flow charts of the data processing method provided by the present invention;

[0023] Figure 2 This is the second flow chart of the data processing method provided by the present invention;

[0024] Figure 3 This is the third flow chart of the data processing method provided by the present invention;

[0025] Figure 4 It is a structural schematic diagram of a data processing device provided by the present invention;

[0026] Figure 5 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0027] In order to make the purpose, technical solutions and advantages of this application clearer, the technical solutions in this application will be clearly and completely described below in conjunction with the drawings in this application. Obviously, the described embodiments are part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0028] It should be noted that, in the embodiments of the present application, words such as "exemplary" or "for example" are used to indicate examples, illustrations or descriptions. Any embodiment or design described as "exemplary" or "for example" in the embodiments of the present application should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of words such as "exemplary" or "for example" is intended to present related concepts in a specific way.

[0029] It should be noted that, in this article, the terms "comprise", "include" or any other variant thereof are intended to cover non-exclusive inclusion, so that the process, method, article or device including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise one..." do not exclude the presence of other identical elements in the process, method, article or device including the element. In addition, it should be pointed out that the scope of the methods and devices in the embodiments of the present application is not limited to performing functions in the order shown or discussed, and may also include performing functions in a substantially simultaneous manner or in reverse order according to the functions involved, for example, the described method may be performed in an order different from that described, and various steps may also be added, omitted, or combined. In addition, the features described with reference to certain examples may be combined in other examples.

[0030] In order to facilitate the clear description of the technical solutions of the embodiments of the present application, in the embodiments of the present application, the words "first", "second", etc. are used to distinguish the same items or similar items with basically the same functions and effects. Those skilled in the art can understand that the words "first", "second", etc. are not limiting the quantity and execution order.

[0031] The embodiments of the present application describe some exemplary embodiments for the purpose of explanation. It should be understood that the present application can be implemented in other ways that are not specifically shown in the drawings.

[0032] like Figure 1 As shown, the embodiment of the present application provides a data processing method, which can be applied to a data processing device. The data processing method may include S101-S106:

[0033] S101: A data processing device obtains an image to be processed.

[0034] Optionally, the image to be processed may include at least one object, and the at least one object may be a person, an animal, a plant, an object, etc.

[0035] S102: The data processing device inputs the image to be processed into a single image-text model to obtain corresponding text description information, and inputs the image to be processed into an image segmentation model to obtain object category information in the image to be processed.

[0036] Optionally, the single image-text model is used to generate as detailed a text description of the image to be processed as possible, which includes a description of the relationship between multiple objects in the image to be processed. The image segmentation model is used to locate and crop the main object in the image to be processed and obtain the category information of the main object.

[0037] For example, Figure 2 As shown, the data processing device can input the image to be processed into the single image-text model to obtain the corresponding text description information "truck passes by pedestrians", and input the image to be processed into the image segmentation model to obtain the object category information in the image to be processed, and the object category information includes "truck" and "pedestrians".

[0038] Optionally, the single image and text model may be GPT-4V, LLaVA, etc., and the image segmentation model may be a pre-trained segmentation model, such as MaskFormer.

[0039] S103: The data processing device inputs the text description information and the object category information into a large language model to generate multi-object related question and answer information.

[0040] The multi-object related question-answer information includes image understanding information and object location information.

[0041] Specifically, the data processing device can input text description information and object category information into the large language model. The large language model can ask and answer several questions related to multiple objects from the perspective of understanding and positioning tasks based on the text description information and object category information, so as to generate multi-object related question and answer information including understanding questions and positioning questions. For understanding questions, the large language model can find answers from the above text description information and output standard answers; for positioning questions, the large language model can use object categories and provide standard answers through the prediction results of the segmentation model.

[0042] Optionally, the control instruction of the above large language model can be: "You will get three text inputs, which are the description information of an image and two object categories. Please ask and answer several questions based on the input from the perspectives of understanding task and positioning task. The requirements must include the image to be processed and the two objects in it."

[0043] For example, Figure 2 As shown, the data processing device can input the text description information "truck passed by pedestrians", object category information "truck" and object category information "pedestrian" into the large language model to obtain multi-object related question and answer information, which can be: "What is the relationship between the truck and pedestrians in the image", "Please locate the pixel-level position of the pedestrian from the image".

[0044] S104: The data processing device performs image backfilling on the objects in the multi-object related question and answer information to obtain first image and text data.

[0045] Specifically, the data processing device can perform image backfill on the objects in the multi-object related question and answer information to obtain the first graphic data, that is, update the text "truck" in the multi-object related question and answer information to the image of the truck, and update the text "pedestrian" in the multi-object related question and answer information to the image of the pedestrian.

[0046] It should be noted that, through the above S101-S104, the data processing device can automatically generate multi-image and text understanding and positioning data in batches, thereby reducing the human cost of generating preliminary data.

[0047] S105. The data processing device inputs the image to be processed and the multi-object associated question and answer information into a reward model to obtain a check item score.

[0048] Optionally, before inputting the image to be processed and the multi-object related question and answer information into the reward model to obtain the inspection item score, the data processing device can obtain training data, where the training data is data obtained after sentence-level annotation processing of the multi-object related question and answer information; and fine-tune the visual language model based on the training data to obtain the reward model.

[0049] Specifically, the quality of the first automatically generated image and text data is difficult to guarantee, and there may be errors and misleading information, which will affect the performance of the model. To solve this problem, a small amount of "multi-object related question and answer information" can be manually labeled for quality, and these labeled data can be classified into fine-grained error types at the sentence level, such as for each sentence containing errors, the annotator selects inaccurate categories from a predefined list. Inaccurate categories can include object hallucinations, attribute errors, object relationship errors, object position errors, and unreasonable image guesses. In addition, the annotator will also evaluate the creativity of the sentence and the level of detail of the overall description. Then, a dedicated visual language model is fine-tuned using these manually labeled data as a reward model. The reward model is trained to output a string that encodes the scores corresponding to various check items when given the image features to be processed and the automatically generated multi-object related question and answer information as input. For example, whether the object is correct, whether the attribute relationship is correct, whether the position is correct, etc. The reward model can provide sentence-level human evaluation scores.

[0050] Additionally, annotators can provide judgments on the creativity of sentences. Creativity means whether the sentence gives a thoughtful and reasonable interpretation or extrapolation of the image. Finally, annotators rate the level of detail of the overall description. These requirements summarize the overall goal of this method: to increase the detail of the text description as much as possible while maintaining the correctness of the relationship description and scene description.

[0051] It should be noted that object hallucination refers to the mention of an object in the text description that does not exist in the image, for example, there is only a dog in the image, but the generated text description mentions a cat. Attribute error refers to an inaccurate description of the attributes of an object, such as a dog in the image is black, but the text says that the dog is white. Object relationship error refers to an incorrect description of the relationship between multiple objects, such as a dog chasing a cat in the image, but the text says that the cat is chasing the dog. Object position error refers to an inaccurate description of the location of the object, such as a dog on the left in the image, but the text mentions that the dog is on the right. Unreasonable image guessing refers to an unreasonable or inconsistent description of the image content, such as a dog on the grass in the image, but the text mentions that the dog is dancing.

[0052] Through fine-grained annotation and manual evaluation, the quality of the data can be effectively improved, ensuring that the text description is highly consistent with the image content.

[0053] Alternatively, if Figure 3 As shown, the data processing device can use the trained reward model to score all input data to obtain the inspection item score.

[0054] Optionally, the input data of the visual language model includes a first sentence and a sentence to be scored, wherein the first sentence includes context information of the sentence to be scored.

[0055] Specifically, in order to provide the necessary reference context for scoring each sentence, the data processing device can add all previously generated texts, i.e., the first sentence, before the sentence to be scored, and explicitly ask the reward model to score the last sentence in a given paragraph. For example, the control instruction of the reward model can be "evaluate the accuracy of the last sentence of this image in the following description and return a number." Due to the fine-grained annotations provided by the annotators, these prompts either lead to positive responses (when no errors are found in the sentence) or detailed negative responses (one or more error types are found in the sentence).

[0056] S106: The data processing device determines whether to determine the first graphic data as multimodal training data according to the inspection item score.

[0057] Alternatively, if Figure 3 As shown, the data processing device can determine the first graphic data as multimodal training data when the inspection item score is greater than the first threshold, that is, select text data without errors or data with only a few erroneous sentences in the whole paragraph as the final high-quality data.

[0058] Optionally, before determining the first graphic data as multimodal training data, the data processing device may also input the first graphic data into a data refinement module to obtain processed first graphic data; wherein the data refinement module is used to remove erroneous data in the first graphic data.

[0059] Specifically, Figure 3 As shown, after selecting the data with higher scores through the reward model, the data processing device can further polish the selected data to form the final high-quality data set. For each sentence in the description, if the reward module believes that it is wrong, it can be deleted from the text description. For example, if the text description information is: "The image shows a white Great Wall car parked on the sidewalk. A pedestrian is walking in front of the car." If the reward module recognizes that the pedestrian is walking in front of the car as an incorrect description of the relationship between multiple objects, then the sentence "A pedestrian is walking in front of the car." will be removed. In the end, the sentence will not be backfilled through the image to form a multi-image question and answer pair, so that the correctness of the constructed data can be further guaranteed.

[0060] Optionally, the error categories of the first graphic data include object hallucination, attribute error, object relationship error, object position error and unreasonable image guessing.

[0061] In the embodiment of the present application, since the generated first image and text data can be scored and screened through the reward model, it can not only improve the data screening efficiency but also improve the description accuracy of the multimodal training data, thereby improving the quality of the training data and providing a solid data foundation for training multimodal large language models for multi-image and text scenarios.

[0062] The above mainly introduces the solution provided by the embodiment of the present application from the perspective of the method. In order to realize the above functions, it includes hardware structures and / or software modules corresponding to the execution of each function. Those skilled in the art should easily realize that, in combination with the units and algorithm steps of each example described in the embodiment disclosed herein, the embodiment of the present application can be implemented in the form of hardware or a combination of hardware and computer software. Whether a function is executed in the form of hardware or computer software driving hardware depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to exceed the scope of the present application.

[0063] The data processing method provided in the embodiment of the present application can be executed by a data processing device or a control module for data processing in the data processing device. In the embodiment of the present application, the data processing device provided in the embodiment of the present application is described by taking the data processing method executed by the data processing device as an example.

[0064] It should be noted that the embodiment of the present application can divide the data processing device into functional modules according to the above method example. For example, each functional module can be divided corresponding to each function, or two or more functions can be integrated into one processing module. The above integrated module can be implemented in the form of hardware or in the form of software functional modules. Optionally, the division of modules in the embodiment of the present application is schematic and is only a logical function division. There may be other division methods in actual implementation.

[0065] like Figure 4 As shown, an embodiment of the present application provides a data processing device 400. The data processing device 400 includes: an acquisition module 401 and a processing module 402. The acquisition module 401 is used to acquire an image to be processed; the processing module 402 is used to input the image to be processed into a single image-text model to obtain corresponding text description information, and input the image to be processed into an image segmentation model to obtain object category information in the image to be processed; the text description information and the object category information are input into a large language model to generate multi-object related question and answer information, and the multi-object related question and answer information includes image understanding information and object location information; the objects in the multi-object related question and answer information are backfilled with images to obtain first image-text data; the image to be processed and the multi-object related question and answer information are input into a reward model to obtain an inspection item score; and it is determined whether to determine the first image-text data as multimodal training data according to the inspection item score.

[0066] Optionally, the reward model is used to score each inspection item and generate an inspection item score based on the scores of all inspection items; the processing module is used to determine the first graphic data as multimodal training data when the inspection item score is greater than a first threshold.

[0067] According to a data processing device provided by the present invention, the processing module 402 is used to input the first graphic data into a data refinement module to obtain processed first graphic data; wherein the data refinement module is used to remove erroneous data in the first graphic data.

[0068] Optionally, the error categories of the first graphic data include object hallucination, attribute error, object relationship error, object position error and unreasonable image guessing.

[0069] Optionally, before the image to be processed and the multi-object related question and answer information are input into the reward model to obtain the inspection item score, the acquisition module 401 is used to obtain training data, and the training data is data obtained after sentence-level annotation processing of the multi-object related question and answer information; the processing module 402 is used to fine-tune the visual language model based on the training data to obtain the reward model.

[0070] Optionally, the input data of the visual language model includes a first sentence and a sentence to be scored, wherein the first sentence contains context information of the sentence to be scored.

[0071] In the embodiment of the present application, since the generated first image and text data can be scored and screened through the reward model, it can not only improve the data screening efficiency but also improve the description accuracy of the multimodal training data, thereby improving the quality of the training data and providing a solid data foundation for training multimodal large language models for multi-image and text scenarios.

[0072] Figure 5 An example of a physical structure diagram of an electronic device is shown in FIG. Figure 5 As shown, the electronic device may include: a processor 510, a communication interface 520, a memory 530 and a communication bus 540, wherein the processor 510, the communication interface 520 and the memory 530 communicate with each other through the communication bus 540. The processor 510 may call the logic instructions in the memory 530 to execute the data processing method, which includes: obtaining an image to be processed; inputting the image to be processed into a single image-text model to obtain corresponding text description information, inputting the image to be processed into an image segmentation model to obtain object category information in the image to be processed; inputting the text description information and the object category information into a large language model to generate multi-object related question-answer information, the multi-object related question-answer information includes image understanding information and object position information; performing image backfilling on the objects in the multi-object related question-answer information to obtain first image-text data; inputting the image to be processed and the multi-object related question-answer information into a reward model to obtain an inspection item score; and determining whether to determine the first image-text data as multimodal training data according to the inspection item score.

[0073] In addition, the logic instructions in the above-mentioned memory 530 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.

[0074] On the other hand, the present invention also provides a computer program product, which includes a computer program. The computer program can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the data processing methods provided by the above methods, which include: obtaining an image to be processed; inputting the image to be processed into a single image-text model to obtain corresponding text description information, and inputting the image to be processed into an image segmentation model to obtain object category information in the image to be processed; inputting the text description information and the object category information into a large language model to generate multi-object related question and answer information, and the multi-object related question and answer information includes image understanding information and object location information; performing image backfilling on objects in the multi-object related question and answer information to obtain first image-text data; inputting the image to be processed and the multi-object related question and answer information into a reward model to obtain an inspection item score; and determining whether to determine the first image-text data as multimodal training data based on the inspection item score.

[0075] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to execute the data processing method provided by the above-mentioned methods, the method comprising: obtaining an image to be processed; inputting the image to be processed into a single image-text model to obtain corresponding text description information, and inputting the image to be processed into an image segmentation model to obtain object category information in the image to be processed; inputting the text description information and the object category information into a large language model to generate multi-object related question and answer information, the multi-object related question and answer information including image understanding information and object location information; performing image backfilling on objects in the multi-object related question and answer information to obtain first image-text data; inputting the image to be processed and the multi-object related question and answer information into a reward model to obtain an inspection item score; and determining whether to determine the first image-text data as multimodal training data based on the inspection item score.

[0076] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0077] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0078] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A data processing method, characterized in that: include: Get the image to be processed; Inputting the image to be processed into a single image-text model to obtain corresponding text description information, and inputting the image to be processed into an image segmentation model to obtain object category information in the image to be processed; Inputting the text description information and the object category information into a large language model to generate multi-object related question and answer information, wherein the multi-object related question and answer information includes image understanding information and object location information; Performing image backfilling on the objects in the multi-object associated question and answer information to obtain first image and text data; Inputting the image to be processed and the multi-object associated question-answer information into a reward model to obtain a check item score; Determining whether to determine the first graphic data as multimodal training data according to the inspection item score; The reward model is used to score each inspection item and generate an inspection item score based on the scores of all inspection items; Before inputting the image to be processed and the multi-object associated question-answer information into a reward model to obtain the inspection item score, the method further includes: Acquire training data, where the training data is data obtained after performing sentence-level annotation processing on the multi-object related question-answer information; Fine-tuning the visual language model based on the training data to obtain the reward model; the input data of the visual language model includes a first sentence and a sentence to be scored, and the first sentence contains context information of the sentence to be scored.

2. The data processing method according to claim 1, characterized in that: The determining, according to the inspection item score, whether to determine the first graphic data as multimodal training data includes: When the inspection item score is greater than a first threshold, the first graphic data is determined as multimodal training data.

3. The data processing method according to claim 2, characterized in that: Before determining the first graphic data as multimodal training data, the method further includes: Inputting the first graphic data into a data refinement module to obtain processed first graphic data; The data refinement module is used to remove data describing errors in the first graphic data.

4. The data processing method according to claim 3, characterized in that: The error categories of the first graphic data include object hallucination, attribute error, object relationship error, object position error and unreasonable image guessing.

5. A data processing device, characterized in that: include: Acquisition module and processing module; The acquisition module is used to acquire the image to be processed; The processing module is used to input the image to be processed into a single image-text model to obtain corresponding text description information, input the image to be processed into an image segmentation model to obtain object category information in the image to be processed; input the text description information and the object category information into a large language model to generate multi-object related question and answer information, wherein the multi-object related question and answer information includes image understanding information and object location information; Perform image backfilling on the objects in the multi-object associated question and answer information to obtain first image and text data; input the to-be-processed image and the multi-object associated question and answer information into a reward model to obtain an inspection item score; and determine whether to determine the first image and text data as multimodal training data according to the inspection item score; The reward model is used to score each inspection item and generate an inspection item score based on the scores of all inspection items; The acquisition module is used to acquire training data, where the training data is data obtained after performing sentence-level annotation processing on the multi-object related question and answer information; The processing module is used to fine-tune the visual language model based on the training data to obtain the reward model; the input data of the visual language model includes a first sentence and a sentence to be scored, and the first sentence contains the context information of the sentence to be scored.

6. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the data processing method according to any one of claims 1 to 4 is implemented.

7. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the data processing method according to any one of claims 1 to 4 is implemented.

8. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the data processing method according to any one of claims 1 to 4 is implemented.

Images