Data processing method and device, electronic equipment and storage medium

Abstract

The invention provides a data processing method and device, electronic equipment and a storage medium. The method comprises the steps that a first image and problem data are acquired; determining at least one detection frame in the first image based on at least one keyword in the problem data; determining at least one sub-image and position information of the at least one sub-image based on the at least one detection frame; and determining input data of the large language model based on the image vectorization feature of the first image, the image vectorization feature of the at least one sub-image, the position information of the at least one sub-image and the problem data.

Description

Technical Field

[0001] The present disclosure relates to the field of artificial intelligence technology, and in particular to a data processing method, device, electronic device and storage medium. Background Art

[0002] Multimodal large models combine advanced language models and image processing technology to respond to user pictures and text questions. Such models have shown significant potential in research fields such as natural sciences. However, when the images provided by users contain a large number of tables, structured data or patterns, multimodal large models still face challenges in accurately capturing image details, understanding information layout and the relationship between data in the table. Summary of the invention

[0003] The present disclosure provides a data processing method, device, electronic device and storage medium to at least solve the above technical problems existing in the prior art.

[0004] According to a first aspect of the present disclosure, there is provided a data processing method, comprising:

[0005] Acquire a first image and question data;

[0006] Based on at least one keyword in the question data, determine at least one detection frame in the first image; based on the at least one detection frame, determine at least one sub-image and position information of the at least one sub-image;

[0007] Based on the image vectorization features of the first image, the image vectorization features of the at least one sub-image, the position information of the at least one sub-image and the question data, input data of the large language model is determined.

[0008] In the above solution, the step of determining at least one detection frame in the first image based on at least one keyword in the question data; and determining at least one sub-image and position information of the at least one sub-image based on the at least one detection frame includes:

[0009] Based on the at least one keyword, detecting a graphic or text corresponding to the at least one keyword in the first image, and framing the graphic or text with a minimum surrounding graphic to form at least one detection frame;

[0010] Determine that the sub-image surrounded by each detection frame is at least one sub-image corresponding to at least one keyword in the first image;

[0011] Determining position information of the at least one sub-image based on the coordinates of the at least one sub-image;

[0012] The position information includes the relative position between at least one sub-image and the position of at least one sub-image in the first image.

[0013] In the above solution, the step of determining input data of the large language model based on the image vectorization feature of the first image, the image vectorization feature of the at least one sub-image, the position information of the at least one sub-image and the question data includes:

[0014] Inputting the image vectorization feature based on the first image, the image vectorization feature of the at least one sub-image, the position information of the at least one sub-image and the question data into a converter;

[0015] The converter converts the image vectorization features of the first image and the image vectorization features of the at least one sub-image of image type into the first image vectorization features and the at least one sub-image vectorization features of text type;

[0016] The converter determines, based on the position information of the at least one sub-image, an association between the first image vectorized feature and the at least one sub-image vectorized feature;

[0017] Determining target information based on an association between the first image vectorized feature and at least one sub-image vectorized feature;

[0018] Based on the target information and the question data, input data of the large language model is determined.

[0019] In the above scheme, the method further comprises:

[0020] The input data is input into the large language model to obtain answer data corresponding to the question data.

[0021] In the above scheme, the image vectorization feature of the first image or the image vectorization feature of the at least one sub-image is determined based on an image encoder, and before determining the input data of the large language model based on the image vectorization feature of the first image, the image vectorization feature of the at least one sub-image, the position information of the at least one sub-image and the question data, the method further includes: training the image encoder based on a sample image, specifically including:

[0022] Annotating the sample image, wherein the annotation includes at least one subsample image in the sample image and position information of the at least one subsample image;

[0023] Inputting the sample image, the at least one sub-sample image and the position information of the at least one sub-sample image into an image encoder, obtaining an image vectorization feature of the sample image and an image vectorization feature of each sub-sample image in the at least one sub-sample image;

[0024] Inputting the image vectorization feature of the sample image and the image vectorization feature of each subsample image in the at least one subsample image into a feedforward network to obtain a decoded image and at least one sub-decoded image;

[0025] determining a loss function of the image encoder based on the sample image, the decoded image, at least one sub-sample image, and at least one sub-decoded image;

[0026] Parameters of the image encoder are adjusted based on a loss function of the image encoder.

[0027] In the above solution, the determining of the input data of the large language model based on the image vectorization feature of the first image, the image vectorization feature of the at least one sub-image, the position information of the at least one sub-image and the question data is implemented by a converter, and the method further includes: training the converter and the large language model, specifically including:

[0028] Inputting the image vectorization features of the sample image, the image vectorization features of each subsample image in at least one subsample image, and the position information of the at least one subsample image into the converter; the image vectorization features of the sample image and the image vectorization features of each subsample image are obtained based on the trained image encoder;

[0029] The converter converts the image vectorization feature into a text image vectorization feature based on the image vectorization feature of the sample image, the image vectorization feature of each subsample image in at least one subsample image, and the position information of the at least one subsample image; the text image vectorization feature is the input of the large language model;

[0030] The vectorized features of the text image are input into the large language model to obtain a predicted answer output by the large language model.

[0031] In the above solution, the training converter and the large language model further include:

[0032] Determining a loss function of the converter and the large language model based on the predicted answer, the sample image and the labeled answer corresponding to the question sample data corresponding to the sample image;

[0033] Based on the loss functions of the converter and the large language model, weights of the converter and the large language model are adjusted.

[0034] According to a second aspect of the present disclosure, a data processing device is provided, the device comprising:

[0035] An acquisition unit, used for acquiring a first image and question data;

[0036] an identification unit, configured to determine at least one detection frame in the first image based on at least one keyword in the question data; and determine at least one sub-image and position information of the at least one sub-image based on the at least one detection frame;

[0037] A determination unit is used to determine input data of a large language model based on the image vectorization features of the first image, the image vectorization features of the at least one sub-image, the position information of the at least one sub-image and the question data.

[0038] According to a third aspect of the present disclosure, there is provided an electronic device, including:

[0039] at least one processor; and

[0040] a memory communicatively connected to the at least one processor; wherein,

[0041] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method described in the present disclosure.

[0042] According to a fourth aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to cause the computer to execute the method described in the present disclosure.

[0043] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] The above and other objects, features and advantages of the exemplary embodiments of the present disclosure will become readily understood by reading the detailed description below with reference to the accompanying drawings. In the accompanying drawings, several embodiments of the present disclosure are shown in an exemplary and non-limiting manner, in which:

[0045] In the drawings, the same or corresponding reference numerals represent the same or corresponding parts.

[0046] Figure 1 A first optional flow chart of the data processing method provided by the embodiment of the present disclosure is shown;

[0047] Figure 2 A second optional flow chart of the data processing method provided by the embodiment of the present disclosure is shown;

[0048] Figure 3 A third optional flow chart of the data processing method provided by the embodiment of the present disclosure is shown;

[0049] Figure 4 A fourth optional flow chart of the data processing method provided by the embodiment of the present disclosure is shown;

[0050] Figure 5 A schematic diagram of an optional training process of an image encoder provided by an embodiment of the present disclosure is shown;

[0051] Figure 6 A schematic diagram of an optional training process of a training converter and a large language model provided in an embodiment of the present disclosure is shown;

[0052] Figure 7 An optional schematic diagram of a prediction process provided by an embodiment of the present disclosure is shown;

[0053] Figure 8 An optional structural schematic diagram of a data processing device provided by an embodiment of the present disclosure is shown;

[0054] Fig. 9 A schematic diagram of the structure of an electronic device according to an embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0055] In order to make the purpose, features, and advantages of the present disclosure more obvious and easy to understand, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the drawings in the embodiments of the present disclosure. Obviously, the described embodiments are only part of the embodiments of the present disclosure, not all of the embodiments. Based on the embodiments in the present disclosure, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present disclosure.

[0056] In the following description, reference is made to “some embodiments”, which describe a subset of all possible embodiments, but it can be understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0057] In the following description, the terms "first\second" involved are merely used to distinguish similar objects and do not represent a specific ordering of the objects. It can be understood that "first\second" can be interchanged with a specific order or sequence where permitted, so that the embodiments of the present disclosure described herein can be implemented in an order other than that illustrated or described herein.

[0058] Unless otherwise defined, all technical and scientific terms used in this disclosure have the same meaning as those commonly understood by those skilled in the art to which this disclosure belongs. The terms used in this disclosure are only for the purpose of describing the embodiments of this disclosure and are not intended to limit this disclosure.

[0059] It should be understood that in the various embodiments of the present disclosure, the size of the serial number of each implementation process does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present disclosure.

[0060] In related technologies, multimodal large models combine advanced language models and image processing technology to respond to user pictures and text questions. Such models have shown significant potential in research fields such as natural sciences. However, when the images provided by users contain a large number of tables, structured data or patterns, multimodal large models still face challenges in accurately capturing image details, understanding information layout and the relationship between data in the table. Specifically:

[0061] 1) For tables in images, the model may not be able to accurately identify the table’s borders, row and column structure, and data content in the table, resulting in incorrect or incomplete answers when answering questions related to the table.

[0062] 2) The presentation of structured data in images is relatively complex, and large multimodal models find it difficult to quickly understand their logical relationships and meanings, which affects the ability to answer complex questions.

[0063] 3) The diversity and complexity of the patterns also pose difficulties for the model. The model may not be able to correctly interpret the information represented by the patterns, limiting its accuracy and comprehensiveness in answering questions.

[0064] This limits the role that large multimodal models can play in answering complex questions. Therefore, an improved large multimodal model processing system is needed to better handle images containing a large number of tables, structured data or patterns and improve the accuracy and practicality of the model.

[0065] The embodiments of the present disclosure provide a data processing method to at least solve part or all of the above technical problems.

[0066] Figure 1 A first optional flow chart of the data processing method provided by the embodiment of the present disclosure is shown, and will be explained according to each step.

[0067] Step S101, obtaining a first image and question data.

[0068] In some embodiments, the first image may include tables, text, structured data and graphics, and the graphics may include people, animals, plants, buildings, vehicles, astronomical scenes, microscopic worlds, works of art (such as paintings, sculptures), activity scenes, etc. The question data may include at least one keyword.

[0069] In some embodiments, there is a correlation between the question data and the first image, and the correlation includes that there is a correspondence between at least one keyword included in the question data and the data or sub-image included in the first image, wherein the sub-image is a partial image in the first image; the correspondence may include a text correspondence, that is, the two are identical or semantically identical (e.g., if the keyword and the data are identical, then there is a correspondence between the keyword and the data), for example, if the keyword includes data XXX, and the first image also includes data XXX, then there is a correspondence between the keyword and the sub-image; the correspondence may also include an image correspondence, that is, the image corresponding to the keyword is the sub-image, for example, if the keyword is "flower", then the first image includes a sub-image of flower; or, if the keyword is "table", then the first image includes a sub-image of table.

[0070] In some embodiments, the question data may include question data related to data or sub-images corresponding to the keywords in the first image, such as specific values ​​of the data, positions, colors, quantities, and sizes of sub-images, etc.

[0071] In some embodiments, a carrier implementing the data processing method (hereinafter referred to as carrier) receives the first image and question data input by the user. The carrier can be a computer program, electronic circuit, database, mobile application, electronic device, cloud computing platform, distributed system, artificial intelligence framework, mathematical model, automation tool and microcontroller, etc., which can implement the algorithm, method flow, software or hardware.

[0072] Step S102: determining at least one detection frame in the first image based on at least one keyword in the question data; and determining at least one sub-image and position information of the at least one sub-image based on the at least one detection frame.

[0073] In some embodiments, the carrier obtains at least one keyword in the problem data, detects the first image based on the at least one keyword, determines at least one detection frame in the first image based on the detection result; confirms that the sub-image surrounded by the at least one detection frame is at least one sub-image; and determines the position information of the at least one sub-image based on the coordinates of the vertices of the at least one sub-image. The position information includes the position information of the sub-image in the first image and the position information of the sub-image relative to the first image; if there are multiple sub-images, the position information also includes the relative position information between each sub-image (such as up and down, left and right, occluded, adjacent, etc.). In the disclosed embodiment, multiple refers to greater than or equal to 2.

[0074] Step S103: Determine input data of a large language model based on the image vectorization features of the first image, the image vectorization features of the at least one sub-image, the position information of the at least one sub-image and the question data.

[0075] In some embodiments, the image vectorization feature is a vectorization feature obtained after mapping the image to the feature space. Correspondingly, the image vectorization feature of the first image is obtained after mapping the first image to the feature space, and the image vectorization feature of at least one sub-image is obtained after mapping the sub-images to the feature space respectively.

[0076] In some embodiments, the input data of the large language model is used to obtain answer data of the large language model for the question data.

[0077] In this way, through the data processing method provided by the embodiment of the present disclosure, in the case where the first image includes a table, the border of the table in the image is pre-identified by detecting the first image provided by the user, and a sub-image corresponding to the keyword in the question data is obtained, and the row and column structure of the table is represented by the position information of the sub-image, and the data content in the table is represented by the feature information of the sub-image, so that the big language can give complete and accurate answers when facing questions related to the table; in the case where the first image includes structured data, the first image provided by the user is detected, and the logical relationship and meaning between the structured data are pre-identified, thereby improving the processing ability of the big language model in the face of structured data problems; in the case where the first image includes diverse and complex graphics, the first image is detected to obtain the corresponding sub-image, and the position information and image vectorization features of the sub-image are used to help the big language model interpret the information represented by each graphics in the first image and the logical relationship between them, thereby improving the accuracy and comprehensiveness of the big language model in answering graphic questions.

[0078] Figure 2A second optional flow chart of the data processing method provided in the embodiment of the present disclosure is shown and will be explained according to each step.

[0079] Step S201, obtaining a first image and question data.

[0080] The specific step flow of step S201 is the same as that of step S101 and will not be repeated here.

[0081] Step S202: Detect the first image based on at least one keyword in the question data to determine at least one detection frame.

[0082] In some embodiments, the carrier performs grammatical analysis on the question data to determine the subject, predicate, object, attributive and adverbial, and determines at least one keyword based on the grammatical analysis results; or the carrier performs word segmentation on the question data, removes stop words, modal particles, etc., and determines at least one keyword based on part-of-speech screening and word frequency statistics.

[0083] In some embodiments, after determining at least one keyword, the carrier detects the first image and obtains text or graphics corresponding to the at least one keyword in the first image. For example, if the keyword includes "little girl", the carrier detects data with the text "little girl" or a graphic presented as "little girl" in the first image.

[0084] In some optional embodiments, for the first keyword, if the first image includes a plurality of corresponding texts or images, the carrier determines the text or image corresponding to the first keyword in combination with other keywords.

[0085] Specifically, in response to the first image including a graphic, the carrier can detect the first image based on each keyword of the question data to obtain multiple detection frames; determine at least one detection frame corresponding to each keyword, perform semantic analysis on the question data, and retain multiple detection frames corresponding to the same subject as the detection frames corresponding to each keyword. Alternatively, the carrier performs semantic analysis on the question data, retains the detection frames that have a one-to-one relationship with the keywords, and other detection frames are candidate detection frames; confirm that among the candidate detection frames, the detection frame with the shortest distance to the reserved detection frame, and the detection frame that overlaps or overlaps with the reserved detection frame is the target detection frame in the candidate detection frame.

[0086] For example, the first image is an image of multiple little girls in a bush of flowers, including multiple little girls and multiple flowers. If the question data is "What color is the flower in the hand of the little girl in blue clothes?", the keywords include blue clothes, little girl, hand and flower; multiple detection frames can be obtained by detecting "little girl", but one detection frame may be obtained by detecting "blue clothes", then the detection frame corresponding to the multiple candidate detection frames of "little girl" that overlaps or overlaps with the detection frame corresponding to "blue clothes" is determined to be the target detection frame corresponding to "little girl". Further, the first image includes multiple hands and multiple flowers, and multiple detection frames will be obtained based on "hands" and "flowers". Then, among the multiple detection frames of "hands", the detection frame that is closest to or overlaps with the detection frame corresponding to "blue clothes" and the detection frame that is closest to or overlaps with the target detection frame corresponding to "little girl" is determined to be the target detection frame of "hand", and the detection frame that is closest to or overlaps with the target detection frame of "hand" among the candidate detection frames of "flower" is determined to be the target detection frame of "flower".

[0087] In some optional embodiments, in response to the first image including a table or structured data, the carrier detects the first image based on the keywords included in the question data, obtains a detection frame corresponding to each keyword, determines a row identifier and a column identifier corresponding to each detection frame, and determines a target detection frame based on the row identifiers and column identifiers of the multiple detection frames. One of the row identifier and the column identifier of the target detection frame is the same as the row identifier or the column identifier corresponding to any detection frame, and the data surrounded by the target detection frame is not empty.

[0088] For example, the first image includes a table of contacts and contact information, in which the first column includes name, contact address, contact number, email address and identity, and the second column includes position (such as chairman's secretary) and data corresponding to the first column, such as the second column includes Zhang San, AA province aa city, 13712345678, 12345@email.com, iD12345; the question data includes "What is the contact number of the chairman's secretary", then the keywords include "chairman's secretary" and "contact number", confirm that the row identifier and column identifier of the chairman's secretary are a1 and b1 respectively, and determine that the row identifier and column identifier of the contact number are a2 and b2 respectively, then determine that the detection box with non-empty data in (a1, b2) or (a2, b1) is the target detection box, that is, the specific contact number of the chairman's secretary.

[0089] Step S203: determining at least one sub-image and position information of at least one sub-image based on the at least one detection frame.

[0090] In some embodiments, the carrier determines the relative position between the sub-images and the position of the sub-image in the first image based on the sub-images corresponding to each detection frame and the coordinates of the sub-images.

[0091] Step S204: Determine input data of a large language model based on the image vectorization features of the first image, the image vectorization features of the at least one sub-image, the position information of the at least one sub-image and the question data.

[0092] In some embodiments, the carrier obtains image vectorization features of a first image and image vectorization features of at least one sub-image, respectively; and determines input data of a large language model based on the image vectorization features of the first image, the image vectorization features of at least one sub-image, position information of the at least one sub-image and the question data.

[0093] In some embodiments, the carrier inputs the image vectorization features based on the first image, the image vectorization features of the at least one sub-image, the position information of the at least one sub-image and the problem data into the converter; the converter converts the image vectorization features of the first image of image type and the image vectorization features of the at least one sub-image into the first image vectorization features and at least one sub-image vectorization features of text type; that is, the converter is used to align the image vectorization features of the image type into the vectorization features of the text type.

[0094] In some embodiments, the converter determines the association between the first image vectorized feature and at least one sub-image vectorized feature, and / or the association between at least one sub-image vectorized feature based on the position information of the at least one sub-image; and determines the target information based on the association between the first image vectorized feature and at least one sub-image vectorized feature, and / or the association between at least one sub-image vectorized feature. The association may include a logical association or a positional association, the logical association includes a clear correspondence relationship or a specific pointing relationship, such as contact information and a specific telephone number; the positional association includes a positional relationship, such as at least one of up and down, left and right, adjacent or overlapping.

[0095] In some optional embodiments, the carrier determines the target information based on the association between the first image vectorized feature and at least one sub-image vectorized feature, which may include determining the target information in the first image based on the logical association or position association between the first imageized feature and at least one sub-image vectorized feature, wherein the target information includes information corresponding to a keyword in the question data, that is, information obtained from the first image based on the keyword.

[0096] In some embodiments, the carrier determines image input data based on the target information, and determines input data of the large language model based on the image input data and the question data. The image input data includes preset answer data detected and recognized in the first image based on the question data.

[0097] Step S205: input the input data into the large language model to obtain answer data corresponding to the question data.

[0098] In some embodiments, the carrier inputs the input data into a large language model, and obtains the output of the large language model as answer data for the question data.

[0099] In this way, through the data processing method provided by the embodiment of the present disclosure, in the case where the first image includes a table, the border of the table in the image is pre-identified by detecting the first image provided by the user, and a sub-image corresponding to the keyword in the question data is obtained, and the row and column structure of the table is represented by the position information of the sub-image, and the data content in the table is represented by the feature information of the sub-image, so that the big language can give complete and accurate answers when facing questions related to the table; in the case where the first image includes structured data, the first image provided by the user is detected, and the logical relationship and meaning between the structured data are pre-identified, thereby improving the processing ability of the big language model in the face of structured data problems; in the case where the first image includes diverse and complex graphics, the first image is detected to obtain the corresponding sub-image, and the position information and image vectorization features of the sub-image are used to help the big language model interpret the information represented by each graphics in the first image and the logical relationship between them, thereby improving the accuracy and comprehensiveness of the big language model in answering graphic questions.

[0100] Figure 3 A third optional flow chart of the data processing method provided in the embodiment of the present disclosure is shown and will be explained according to each step.

[0101] Step S301: training an image encoder based on a sample image.

[0102] In some embodiments, the image encoder is used to convert the image into a feature space to obtain an image feature vector; the training goal of the image encoder is to make the image feature vector converted by the image encoder capture the essential features and key information of the image as accurately as possible, and generate an image feature vector with high accuracy and strong representativeness, so as to better play a role in subsequent tasks.

[0103] In some embodiments, a sample image is annotated in advance, and the annotation includes at least one sub-sample image in the sample image. The sample image and at least one sub-sample image are input into an image encoder to obtain image vectorization features of the sample image and image vectorization features of each sub-sample image.

[0104] The image vectorization features of the sample image and the image vectorization features of each subsample image in the at least one subsample image are input into a feedforward network to obtain a decoded image and at least one sub-decoded image. Based on the sample image, the decoded image, the at least one subsample image and the at least one sub-decoded image, a loss function of the image encoder is determined; and the parameters of the image encoder are adjusted based on the loss function of the image encoder. Optionally, the loss function of the image encoder can be determined by means of cross entropy. The feedforward network is used to map the image vectorization features in the feature space to an image.

[0105] Specifically, an image encoder is used to map an image to a feature space, and then a feedforward network is used to map the image vectorized features in the feature space back to the image. The higher the degree of training of the image encoder, the smaller the difference between the sample image and the decoded image, or between the sub-sample image and the sub-decoded image, which also indicates that the mapping function of the image encoder is more accurate.

[0106] Step S302: training the converter based on the sample image.

[0107] In some embodiments, after the image encoder is trained based on step S301, the parameters (or weights) of the image encoder are frozen (ie, the parameters of the image encoder are no longer changed or altered), and the converter is trained based on the output of the image encoder.

[0108] In some embodiments, the converter is used to align image vectorization features of image type into image vectorization features of text type.

[0109] In some embodiments, the carrier obtains the image vectorization features of the sample image and the image vectorization features of each subsample image in at least one subsample image based on the trained image encoder; the image vectorization features of the sample image, the image vectorization features of each subsample image in at least one subsample image, and the position information of the at least one subsample image are input into the converter. The converter converts the image vectorization features of the sample image and the image vectorization features of each subsample image in at least one subsample image into text-type image vectorization features, and confirms the target sample information based on the position information of the at least one subsample image; and determines the input data sample of the large language model based on the target sample information and the problem data sample.

[0110] The vectorized features of the text image are input into the large language model to obtain a predicted answer output by the large language model; based on the predicted answer and the labeled answer corresponding to the sample image and the question sample data corresponding to the sample image, the loss function of the converter and the large language model is determined; based on the loss function of the converter and the large language model, the weights of the converter and the large language model are adjusted.

[0111] Optionally, the converter can also convert the vectorized features of the image into text. The converter then converts the image vectorized features of the sample image and the image vectorized features of each subsample image in at least one subsample image into text, and based on the position information of the at least one subsample image, determines the target sample information in the text; based on the target sample information and the problem data sample, determines the input data sample of the large language model.

[0112] Step S303, obtaining the first image and question data.

[0113] Step S304: detect the first image based on at least one keyword in the question data to determine at least one detection frame.

[0114] Step S305: determine at least one sub-image and position information of the at least one sub-image based on the at least one detection frame.

[0115] Step S306: Determine input data of a large language model based on the image vectorization features of the first image, the image vectorization features of the at least one sub-image, the position information of the at least one sub-image and the question data.

[0116] Step S307: input the input data into the large language model to obtain answer data corresponding to the question data.

[0117] The specific step flow from step S303 to step S307 is the same as that from step S201 to step S205, and will not be repeated here.

[0118] It should be noted that, in step S306, the image vectorization features of the first image and the image vectorization features of the at least one sub-image are obtained based on the image encoder trained in step S301; that is, the image vectorization features of the first image are input into the trained image encoder to obtain the image vectorization features of the first image; and each of the at least one sub-image is respectively input into the image encoder to obtain the image vectorization features corresponding to each sub-image.

[0119] Step S306 is implemented based on the converter trained in step S302, that is, the image vectorization features of the first image, the image vectorization features of the at least one sub-image and the position information of the at least one sub-image are input into the converter to obtain target information, and the input data of the large language model is determined based on the target information and the problem data.

[0120] In this way, through the data processing method provided by the embodiment of the present disclosure, in the case where the first image includes a table, the border of the table in the image is pre-identified by detecting the first image provided by the user, and a sub-image corresponding to the keyword in the question data is obtained, and the row and column structure of the table is represented by the position information of the sub-image, and the data content in the table is represented by the feature information of the sub-image, so that the big language can give complete and accurate answers when facing questions related to the table; in the case where the first image includes structured data, the first image provided by the user is detected, and the logical relationship and meaning between the structured data are pre-identified, thereby improving the processing ability of the big language model in the face of structured data problems; in the case where the first image includes diverse and complex graphics, the first image is detected to obtain the corresponding sub-image, and the position information and image vectorization features of the sub-image are used to help the big language model interpret the information represented by each graphics in the first image and the logical relationship between them, thereby improving the accuracy and comprehensiveness of the big language model in answering graphic questions.

[0121] Figure 4 A fourth optional flow chart of the data processing method provided in the embodiment of the present disclosure is shown and will be explained according to each step.

[0122] In this embodiment, the first image includes a table as an example for description; those skilled in the art should understand that when the first image includes graphics or structured data, the processing method is similar to that when the first image includes a table.

[0123] Step S401, training the image encoder.

[0124] In some embodiments, the image encoder is used to obtain image vectorization features of the sample image and image vectorization features of at least one sub-sample image based on position information of the sample image and at least one sub-sample image in the sample image.

[0125] Figure 5 A schematic diagram of an optional training process of an image encoder provided by an embodiment of the present disclosure is shown.

[0126] like Figure 5As shown, based on the keywords in the problem sample data, the sample image is detected to determine at least one detection frame; the sub-image enclosed by each detection frame in the at least one detection frame is determined to be a sub-sample image, and the position information of each sub-sample image is determined based on the detection frame.

[0127] Optionally, the position information may include coordinates of any two diagonal corners of the detection frame, such as the coordinates of the upper left corner and the lower right corner, or the coordinates of the upper right corner and the lower left corner. Figure 5 Based on the keyword detection in the problem sample data, two detection boxes are obtained, namely region1 and region2. The location information of region1 is (x1, y1, x2, y2), where (x1, y1) is the coordinate of the upper left corner of region1, and (x2, y2) is the coordinate of the lower right corner of region1; if (x1, y1) is the coordinate of the lower left corner of region1, then (x2, y2) is the coordinate of the upper right corner of region1; the location information of region2 is (x3, x4, y3, y4). According to the coordinates of region1 and region2, region1 is above region2. Optionally, x1 and x3 are the same or different (if the left borders of region1 and region2 are aligned, then x1 and x3 are the same, and if the left borders of region1 and region2 are not aligned, then x1 and x3 are different).

[0128] In some embodiments, the image encoder may be a CLIP model; the image encoder is also used to identify the relative position of the detection frame based on the position information of the subsample image, specifically including: the relative position of each detection frame in the sample image; when there are more than two detection frames, and one of the relative positions between the detection frames (such as region1 is above region2).

[0129] In some embodiments, based on the keywords of the question data, the sample image ( Figure 5 The carrier inputs the position information of the sample image and each sub-sample image into the image encoder, and the image encoder determines the sub-sample image based on the position information of each sub-sample image, converts the sample image and each sub-sample image into the feature space, and obtains the image vectorization feature ( Figure 5 image embedding) and image vectorization features of subsample images ( Figure 5 in region1embedding and region2 embedding).

[0130] The image vectorization features of the sample image and the image vectorization features of the sub-sample image are respectively input into a feedforward network (or decoder) such as FNN to obtain a decoded image and a decoded sub-image, and the loss function of the image encoder is determined based on the sample image, the sub-sample image, the decoded image and the decoded sub-image, and the weight of the image encoder is adjusted based on the loss function.

[0131] Step S402, training the converter and the large language model.

[0132] Figure 6 A schematic diagram of an optional training process for a training converter and a large language model provided in an embodiment of the present disclosure is shown.

[0133] like Figure 6 As shown, when training the converter (Q-former) and the large language model (LLM), the trained image encoder is frozen.

[0134] The image vectorization features of the sample image and the sub-sample image output by the image encoder, as well as the position information of each sub-sample image, are input into the converter, which converts the image vectorization features of the sub-sample image and the sample image into text, determines the layout information of the text based on the position information of each sub-sample image, and determines the target information based on the layout information and the text.

[0135] by Figure 6 For example, region1 is the secretary of the board of directors, and region2 is the specific contact number. According to the location information, region1 is above region2. When determining the target information, region1 is before region2, that is, the target information is " <region1>Secretary to the Board <region2>+86-10-80000000", the input sample of the large language model is determined based on the target information and the question data sample, that is:

[0136] " <region1>Secretary to the Board <region2>+86-10-80000000

[0137] Input query: What is the contact number of the board secretary? "

[0138] The input sample of the large language model is input into the large language model to obtain a predicted answer; based on the predicted answer and the labeled answer (ie, "the contact number is +86-10-80000000"), the parameters of the converter and the large language model are adjusted.

[0139] Step S403: predicting the problem data.

[0140] Figure 7 An optional schematic diagram of the prediction process provided by an embodiment of the present disclosure is shown.

[0141] The question data is "What is the contact number of the board secretary?", the first image is Figure 7 The table in (i.e., input image) is used as an example to illustrate.

[0142] In some embodiments, the carrier determines that the keywords in the question data are "board secretary" and "contact number", detects the first image based on the keywords, and obtains region1 (including the detection frame of the board secretary) and region2 (including the detection frame of the specific contact number). Based on the detection frame, a sub-image is determined, that is, the image surrounded by the detection frame is the sub-image.

[0143] In some optional embodiments, when the carrier performs detection, what is detected may be "board secretary" and "contact phone number", and based on the row identifier and column identifier of the two, a detection frame of the specific contact phone number, namely region2, is determined.

[0144] In some embodiments, the carrier inputs position information of the first image and at least one sub-image into an image encoder to obtain image vectorization features of the first image and image vectorization features corresponding to each sub-image.

[0145] The image vectorization features of the first image, the image vectorization features corresponding to each sub-image, and the position information of each sub-image are input into the converter to obtain the target information, that is, " <region1>Secretary to the Board <region2>+86-10-80000000". Based on the target information and question data, the input data of the large language model is determined and input into the large language model to obtain the answer data "The contact number is +86-10-80000000".

[0146] In this way, according to the data processing method provided by the embodiment of the present disclosure, based on the user's behavioral habits of querying table data and exploring answers, auxiliary learning tasks are designed, and supervised learning strategies are used to enhance the model's ability to extract key information. In the training stage, a phased training strategy is adopted. First, the image encoder is trained to capture the layout information more accurately; then, the image encoder is frozen, and the converter and the large language model are trained to train the LLM's ability to integrate layout information. Through a rule-driven automated process, the data required for supervised learning tasks can be quickly constructed while ensuring the accuracy of data construction. Compared with the current multimodal large model based on document question and answer, the fine-tuning training of the image encoder in the scheme is more targeted and extracts layout structure information. Compared with the method that relies on text descriptions to obtain structural information, it does not require a large model to reconstruct position information. It is more direct and reliable to extract information directly from the image, reducing the possibility of information loss.

[0147] Figure 8 An optional structural schematic diagram of a data processing device provided in an embodiment of the present disclosure is shown, and will be described according to each part.

[0148] In some embodiments, the data processing device 700 includes an acquisition unit 701 , an identification unit 702 , and a determination unit 703 .

[0149] The acquisition unit 701 is used to acquire the first image and question data;

[0150] The identification unit 702 is configured to determine at least one detection frame in the first image based on at least one keyword in the question data; and determine at least one sub-image and position information of the at least one sub-image based on the at least one detection frame;

[0151] The determination unit 703 is used to determine input data of the large language model based on the image vectorization features of the first image, the image vectorization features of the at least one sub-image, the position information of the at least one sub-image and the question data.

[0152] The recognition unit 702 is specifically configured to detect a graphic or text corresponding to the at least one keyword in the first image based on the at least one keyword, and frame the graphic or text with a minimum surrounding graphic to form at least one detection frame;

[0153] Determine that the sub-image surrounded by each detection frame is at least one sub-image corresponding to at least one keyword in the first image;

[0154] Determining position information of the at least one sub-image based on the coordinates of the at least one sub-image;

[0155] The position information includes the relative position between at least one sub-image and the position of at least one sub-image in the first image.

[0156] The determining unit 703 is specifically configured to input the image vectorization feature based on the first image, the image vectorization feature of the at least one sub-image, the position information of the at least one sub-image and the problem data into the converter;

[0157] The converter converts the image vectorization features of the first image and the image vectorization features of the at least one sub-image of image type into the first image vectorization features and the at least one sub-image vectorization features of text type;

[0158] The converter determines, based on the position information of the at least one sub-image, an association between the first image vectorized feature and the at least one sub-image vectorized feature;

[0159] Determining target information based on an association between the first image vectorized feature and at least one sub-image vectorized feature;

[0160] Based on the target information and the question data, input data of the large language model is determined.

[0161] The determining unit 703 is further configured to input the input data into the large language model to obtain answer data corresponding to the question data.

[0162] In some embodiments, the data processing device 700 further includes a training unit 704 .

[0163] In some embodiments, the image vectorization feature of the first image or the image vectorization feature of the at least one sub-image is determined based on an image encoder, and before determining the input data of the large language model based on the image vectorization feature of the first image, the image vectorization feature of the at least one sub-image, the position information of the at least one sub-image and the question data, the training unit 704 is used to train the image encoder based on the sample image, specifically including:

[0164] Annotating a sample image, wherein the annotation includes at least one subsample image in the sample image;

[0165] Inputting the sample image and the at least one sub-sample image into an image encoder to obtain an image vectorization feature of the sample image and an image vectorization feature of each sub-sample image in the at least one sub-sample image;

[0166] Inputting the image vectorization feature of the sample image and the image vectorization feature of each subsample image in the at least one subsample image into a feedforward network to obtain a decoded image and at least one sub-decoded image;

[0167] determining a loss function of the image encoder based on the sample image, the decoded image, at least one sub-sample image, and at least one sub-decoded image;

[0168] Parameters of the image encoder are adjusted based on a loss function of the image encoder.

[0169] In some embodiments, the determining the input data of the large language model based on the image vectorization feature of the first image, the image vectorization feature of the at least one sub-image, the position information of the at least one sub-image and the question data is implemented by a converter, and the training unit 704 is also used to train the converter and the large language model, specifically including:

[0170] Inputting the image vectorization features of the sample image, the image vectorization features of each subsample image in at least one subsample image, and the position information of the at least one subsample image into the converter; the image vectorization features of the sample image and the image vectorization features of each subsample image are obtained based on the trained image encoder;

[0171] The converter converts the image vectorization feature into a text image vectorization feature based on the image vectorization feature of the sample image, the image vectorization feature of each subsample image in at least one subsample image, and the position information of the at least one subsample image; the text image vectorization feature is the input of the large language model;

[0172] The vectorized features of the text image are input into the large language model to obtain a predicted answer output by the large language model.

[0173] The training unit 704 is further configured to determine a loss function of the converter and the large language model based on the predicted answer, the sample image and the labeled answer corresponding to the question sample data corresponding to the sample image;

[0174] Based on the loss functions of the converter and the large language model, weights of the converter and the large language model are adjusted.

[0175] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device and a readable storage medium.

[0176] Fig. 9 A schematic block diagram of an example electronic device 800 that can be used to implement an embodiment of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.

[0177] like Fig. 9 As shown, the electronic device 800 includes a computing unit 801, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 802 or a computer program loaded from a storage unit 808 into a random access memory (RAM) 803. In the RAM 803, various programs and data required for the operation of the electronic device 800 can also be stored. The computing unit 801, the ROM 802, and the RAM 803 are connected to each other via a bus 804. An input / output (I / O) interface 805 is also connected to the bus 804.

[0178] Multiple components in the electronic device 800 are connected to the I / O interface 805, including: an input unit 806, such as a keyboard, a mouse, etc.; an output unit 807, such as various types of displays, speakers, etc.; a storage unit 808, such as a disk, an optical disk, etc.; and a communication unit 809, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 809 allows the electronic device 800 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0179] The computing unit 801 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 801 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 801 performs the various methods and processes described above, such as data processing methods. For example, in some embodiments, the data processing method may be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage unit 808. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 800 via ROM 802 and / or communication unit 809. When the computer program is loaded into RAM 803 and executed by the computing unit 801, one or more steps of the data processing method described above may be performed. Alternatively, in other embodiments, the computing unit 801 may be configured to perform the data processing method in any other appropriate manner (e.g., by means of firmware).

[0180] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0181] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0182] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0183] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0184] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.

[0185] A computer system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship of client and server is generated by computer programs running on respective computers and having a client-server relationship with each other. The server may be a cloud server, a server of a distributed system, or a server combined with a blockchain.

[0186] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document does not limit this.

[0187] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include at least one of the features. In the description of the present disclosure, "plurality" means two or more, unless otherwise clearly and specifically defined.

[0188] The above is only a specific embodiment of the present disclosure, but the protection scope of the present disclosure is not limited thereto. Any person skilled in the art who is familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed in the present disclosure, which should be included in the protection scope of the present disclosure. Therefore, the protection scope of the present disclosure should be based on the protection scope of the claims.

Claims

1. A data processing method, the method comprising: Acquire a first image and question data; Determining at least one detection box in the first image based on at least one keyword in the question data; Based on the at least one detection frame, determining at least one sub-image and position information of the at least one sub-image; Based on the image vectorization features of the first image, the image vectorization features of the at least one sub-image, the position information of the at least one sub-image and the question data, input data of the large language model is determined.

2. The method according to claim 1, wherein the at least one detection box in the first image is determined based on at least one keyword in the question data; Determining at least one sub-image and position information of the at least one sub-image based on the at least one detection frame includes: Based on the at least one keyword, detecting a graphic or text corresponding to the at least one keyword in the first image, and framing the graphic or text with a minimum surrounding graphic to form at least one detection frame; Determine that the sub-image surrounded by each detection frame is at least one sub-image corresponding to at least one keyword in the first image; Determining position information of the at least one sub-image based on the coordinates of the at least one sub-image; The position information includes the relative position between at least one sub-image and the position of at least one sub-image in the first image.

3. The method according to claim 1, wherein the step of determining input data of a large language model based on the image vectorization feature of the first image, the image vectorization feature of the at least one sub-image, the position information of the at least one sub-image and the question data comprises: Inputting the image vectorization feature based on the first image, the image vectorization feature of the at least one sub-image, the position information of the at least one sub-image and the question data into a converter; The converter converts the image vectorization features of the first image and the image vectorization features of the at least one sub-image of image type into the first image vectorization features and the at least one sub-image vectorization features of text type; The converter determines, based on the position information of the at least one sub-image, an association between the first image vectorized feature and the at least one sub-image vectorized feature; Determining target information based on an association between the first image vectorized feature and at least one sub-image vectorized feature; Based on the target information and the question data, input data of the large language model is determined.

4. The method according to claim 1, further comprising: The input data is input into the large language model to obtain answer data corresponding to the question data.

5. The method according to claim 1, wherein the image vectorization feature of the first image or the image vectorization feature of the at least one sub-image is determined based on an image encoder, and before determining the input data of the large language model based on the image vectorization feature of the first image, the image vectorization feature of the at least one sub-image, the position information of the at least one sub-image and the question data, the method further comprises: The image encoder is trained based on the sample image, specifically comprising: Annotating a sample image, wherein the annotation includes at least one subsample image in the sample image; Inputting the sample image and the at least one sub-sample image into an image encoder to obtain an image vectorization feature of the sample image and an image vectorization feature of each sub-sample image in the at least one sub-sample image; Inputting the image vectorization feature of the sample image and the image vectorization feature of each subsample image in the at least one subsample image into a feedforward network to obtain a decoded image and at least one sub-decoded image; determining a loss function of the image encoder based on the sample image, the decoded image, at least one sub-sample image, and at least one sub-decoded image; Parameters of the image encoder are adjusted based on a loss function of the image encoder.

6. The method according to claim 1, wherein the determining the input data of the large language model based on the image vectorization feature of the first image, the image vectorization feature of the at least one sub-image, the position information of the at least one sub-image and the question data is implemented by a converter, and the method further comprises: Training transformers and large language models, including: Inputting the image vectorization features of the sample image, the image vectorization features of each subsample image in at least one subsample image, and the position information of the at least one subsample image into the converter; the image vectorization features of the sample image and the image vectorization features of each subsample image are obtained based on the trained image encoder; The converter converts the image vectorization feature into a text image vectorization feature based on the image vectorization feature of the sample image, the image vectorization feature of each subsample image in at least one subsample image, and the position information of the at least one subsample image; the text image vectorization feature is the input of the large language model; The vectorized features of the text image are input into the large language model to obtain a predicted answer output by the large language model.

7. The method according to claim 6, wherein the training converter and the large language model further comprises: Determining a loss function of the converter and the large language model based on the predicted answer, the sample image and the labeled answer corresponding to the question sample data corresponding to the sample image; Based on the loss functions of the converter and the large language model, weights of the converter and the large language model are adjusted.

8. A data processing device, comprising: An acquisition unit, used for acquiring a first image and question data; an identification unit, configured to determine at least one detection frame in the first image based on at least one keyword in the question data; Based on the at least one detection frame, determining at least one sub-image and position information of the at least one sub-image; A determination unit is used to determine input data of a large language model based on the image vectorization features of the first image, the image vectorization features of the at least one sub-image, the position information of the at least one sub-image and the question data.

9. An electronic device, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the method according to any one of claims 1 to 7.

10. A non-transitory computer-readable storage medium storing computer instructions, characterized in that: The computer instructions are used to make a computer execute the method according to any one of claims 1-7.

Images