Data processing method and apparatus

Abstract

The application provides a data processing method and device. The data processing method comprises: obtaining a target image, wherein the target image comprises a to-be-labeled subject task; inputting the target image and a preset prompt text into an image-text conversion model to obtain at least one target subject and subject text corresponding to each target subject output by the image-text conversion model, wherein the preset prompt text is used to guide the image-text conversion model to extract at least one target subject and subject text corresponding to each target subject from the to-be-labeled subject task. According to the application, the labeling efficiency of the to-be-labeled subject task in the image is improved.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to data processing methods. This application also relates to a data processing apparatus, a computing device, a computer-readable storage medium, and a computer program product. Background Technology

[0002] With the continuous development of computer technology, in the field of intelligent education, teaching tasks for various subjects are usually stored in the form of images.

[0003] However, when managing teaching tasks across different subjects, multiple models are typically used in collaboration. For example, a text recognition model is used to identify semantics in an image. Then, a text annotation model is used to annotate the identified semantics. However, this collaborative approach with multiple models can easily introduce cascading errors into the annotation results, leading to low accuracy. Therefore, accurately identifying teaching tasks for each subject has become a pressing issue. Summary of the Invention

[0004] In view of this, embodiments of this application provide a data processing method. This application also relates to a data processing apparatus, a computing device, a computer-readable storage medium, and a computer program product, to address the aforementioned problems existing in the prior art.

[0005] According to a first aspect of the embodiments of this application, a data processing method is provided, including: Acquire a target image, wherein the target image includes a subject task to be labeled; The target image and the preset prompt text are input into the image-to-text conversion model to obtain at least one target subject and the corresponding subject text of each target subject output by the image-to-text conversion model. The preset prompt text is used to guide the image-to-text conversion model to extract at least one target subject and the corresponding subject text of each target subject from the subject task to be labeled.

[0006] According to a second aspect of the embodiments of this application, a data processing apparatus is provided, comprising: An acquisition unit is configured to acquire a target image, wherein the target image includes a subject task to be labeled; The processing unit is configured to input the target image and preset prompt text into an image-to-text conversion model to obtain at least one target subject and the subject text corresponding to each target subject output by the image-to-text conversion model. The preset prompt text is used to guide the image-to-text conversion model to extract at least one target subject and the subject text corresponding to each target subject from the subject task to be labeled.

[0007] According to a third aspect of the embodiments of this application, a computing device is provided, comprising: Memory and processor; The memory is used to store computer programs / instructions, and the processor is used to execute the computer programs / instructions, which, when executed by the processor, implement the steps of the above-described data processing method.

[0008] According to a fourth aspect of the embodiments of this application, a computer-readable storage medium is provided that stores a computer program / instructions, which, when executed by a processor, implement the steps of the above-described data processing method.

[0009] According to a fifth aspect of the embodiments of this application, a computer program product is provided, including a computer program / instructions that, when executed by a processor, implement the steps of the above-described data processing method.

[0010] According to the data processing method provided in this application, a target image containing the subject task to be labeled is obtained, and the target image and preset prompt text are input into an image-to-text conversion model. The image-to-text conversion model identifies the subject task to be labeled in the target image, and extracts at least one target subject from the subject task using the preset prompt text, and determines the corresponding subject text for each target subject. Through the identification and extraction operations within the image-to-text conversion model, no collaboration with other models is required, thus avoiding cascading errors. Furthermore, the preset prompt text guidance enables the image-to-text model to classify and label the subject task according to the preset prompt text, obtaining at least one target subject and the corresponding subject text for each target subject. Therefore, this application avoids cascading errors between multiple model collaborations and improves the efficiency and accuracy of labeling subject tasks by using preset prompt text. Attached Figure Description

[0011] Figure 1 A flowchart of a data processing method according to an embodiment of this application is shown; Figure 2 A flowchart illustrating a data processing method for subject content annotation according to an embodiment of this application is shown. Figure 3 This illustration shows a schematic diagram of the structure of a data processing apparatus according to an embodiment of this application; Figure 4 A structural block diagram of a computing device according to an embodiment of this application is shown. Detailed Implementation

[0012] Many specific details are set forth in the following description to provide a full understanding of this application. However, this application can be implemented in many other ways different from those described herein, and those skilled in the art can make similar extensions without departing from the spirit of this application; therefore, this application is not limited to the specific embodiments disclosed below.

[0013] The terminology used in one or more embodiments of this application is for the purpose of describing particular embodiments only and is not intended to limit the scope of one or more embodiments of this application. The singular forms “a,” “the,” and “the” used in one or more embodiments of this application and in the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” used in one or more embodiments of this application refers to and includes any or all possible combinations of one or more associated listed items.

[0014] It should be understood that although the terms first, second, etc., may be used to describe various information in one or more embodiments of this application, such information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, first may also be referred to as second without departing from the scope of one or more embodiments of this application, and similarly, second may also be referred to as first. Depending on the context, the word "if" as used herein may be interpreted as "when," "when," or "in response to a determination."

[0015] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, use and processing of the relevant data must comply with the relevant laws, regulations and standards of the relevant regions, and corresponding operation entry points are provided for users to choose to authorize or refuse.

[0016] The data processing method disclosed in this application is applied to scenarios involving the recognition of text information in images. For example, it can be applied in intelligent education scenarios to recognize text in images and classify and label the content within the images.

[0017] With the continuous development of computer technology, in intelligent education scenarios, multiple models are often used collaboratively to manage teaching tasks across various subjects. For example, a large model might be used to recognize assignments, while a smaller model is used for classification. The captured images are converted into plain text containing the assignment content by a multimodal large model, and then a smaller model performs multi-label classification on the assignment content. However, this approach has several drawbacks. First, when performing multi-label classification on the assignment content, the recognition performance on plain text is limited, which can easily lead to error accumulation. Second, the maintenance cost of multiple models is too high, and the processing efficiency is low.

[0018] Therefore, this application provides a data processing method. Based on the processing capabilities of an image-to-text conversion model and combined with preset prompt text, this application extracts text from an image and guides the image-to-text conversion model to annotate the extracted text according to the preset prompt text. This achieves text extraction and annotation from an image without requiring the collaboration of two models, thus avoiding cascading errors and improving processing efficiency. This application also relates to a data processing apparatus, a computing device, a computer-readable storage medium, and a computer program product, which will be described in detail in the following embodiments.

[0019] Figure 1 A flowchart of a data processing method according to an embodiment of this application is shown, specifically including the following steps 102-104: Step 102: Obtain the target image, wherein the target image includes the subject task to be labeled.

[0020] The target image can be understood as an image containing subject-specific tasks obtained in a smart teaching scenario, and the image's sharpness assessment result meets a sharpness threshold. The target image can be acquired, for example, by taking a picture with a camera or by downloading it beforehand. It should be understood that this application does not limit the method of acquiring the target image.

[0021] In one specific embodiment provided in this application, a target image containing the subject task to be labeled is obtained from different intelligent teaching scenarios, provided that the sharpness assessment result meets the sharpness threshold.

[0022] Acquiring the target image includes: Acquire an initial image and determine the sharpness assessment result of the initial image, wherein the initial image includes an initial subject task to be labeled, and the sharpness assessment result is determined based on the attribute parameters of the initial subject task to be labeled; If the sharpness assessment result meets the sharpness threshold, the initial image is determined as the target image; If the sharpness assessment result does not meet the sharpness threshold, the attribute parameters of the initial subject task to be labeled are adjusted until the sharpness assessment result of the initial image meets the sharpness threshold, thus obtaining the target image.

[0023] The initial image can be understood as the original image without any quality processing. It is the original data for obtaining the target image and contains the initial subject task to be labeled.

[0024] The sharpness assessment result can be understood as the result obtained after quantitatively analyzing the identifiability of the initial image based on the attribute parameters of the initial subject task to be labeled. It should be understood that the sharpness assessment result involved in this application is used to determine the sharpness of the subject task to be labeled in the image.

[0025] The initial subject task to be labeled can be understood as the original subject task content information contained in the initial image that has not been identified and classified by the image-to-text conversion model.

[0026] Attribute parameters can be understood as image parameters used to determine the recognizability of the initial subject task to be labeled in the initial image. They are used for sharpness assessment and can specifically include parameters related to the subject task to be labeled, such as image resolution, text sharpness, occlusion rate, contrast, brightness, and noise rate.

[0027] A sharpness threshold can be understood as a pre-set critical value for sharpness evaluation to ensure that the image-to-text conversion model can accurately identify the subject task to be labeled and to avoid recognition errors caused by image quality. For example, it can be a score threshold, parameter threshold, or achievement level, and serves as a standard for determining whether the initial image needs optimization. If the sharpness evaluation result reaches or exceeds this threshold, the image is used as the target image. For ease of understanding, the sharpness threshold involved in this application can be at least one of the following: an evaluation score of 80 or above, an unobstructed rate of 90% or above, or an achievement level of excellent or good.

[0028] In one specific embodiment provided in this application, an initial image containing the initial subject task to be labeled is acquired. Then, a preset sharpness evaluation algorithm is used to extract various attribute parameters of the initial subject task to be labeled. For example, resolution, contrast, text sharpness, etc. The initial image is quantitatively analyzed based on the extracted attribute parameters to obtain a sharpness evaluation result. If the sharpness evaluation result of the initial image is greater than or equal to a preset sharpness threshold, the initial image is determined as the target image.

[0029] In this scenario, if the sharpness assessment result of the initial image is less than a preset sharpness threshold, the attribute parameters affecting sharpness in the initial image are located based on the sharpness assessment result. For example, locating attribute parameters affecting the sharpness assessment result could involve assessing unclear objects in the initial image; for example, if brightness is insufficient, locate the brightness parameter; if text is blurry, locate the text sharpness parameter; if there is a lot of noise, locate the noise rate parameter, etc. For the parameters obtained from the located objects, corresponding digital image processing techniques are used for targeted adjustments. These could include brightness enhancement, sharpening, noise reduction, contrast adjustment, and image restoration. The attribute parameters are then extracted again from the adjusted image, and a sharpness assessment is performed until the sharpness assessment result of the initial image meets the sharpness threshold. The final adjusted image is then determined as the target image.

[0030] According to a specific embodiment provided in this application, the sharpness assessment result of the initial image is determined based on the attribute parameters of the subject task to be labeled, so that the final sharpness assessment result can accurately reflect the sharpness of the subject task to be labeled, thereby facilitating the subsequent extraction of the subject task by the image-to-text conversion model. Furthermore, if the sharpness assessment result does not meet the sharpness threshold, specific objects in the initial image that do not meet the sharpness threshold are identified, and then adjusted accordingly so that the adjusted initial image meets the sharpness threshold. This ensures the accuracy of the image-to-text conversion model in recognizing the subject task to be labeled in the target image.

[0031] After obtaining a target image whose sharpness assessment result meets the sharpness threshold, this application inputs it along with preset prompt text into an image-to-text conversion model. Based on the recognition and extraction capabilities of the image-to-text conversion model, at least one target subject contained in the target image and the corresponding subject text for each target subject are obtained. The specific implementation method is shown in step 104.

[0032] Step 104: Input the target image and the preset prompt text into the image-to-text conversion model to obtain at least one target subject and the corresponding subject text of each target subject output by the image-to-text conversion model. The preset prompt text is used to guide the image-to-text conversion model to extract at least one target subject and the corresponding subject text of each target subject from the subject task to be labeled.

[0033] The preset prompt text can be understood as pre-designed text used for classifying and labeling text content. In the image-to-text conversion model, the preset prompt text guides the model to extract at least one subject from the text content in the recognized image and determine the corresponding subject text for each subject. It should be understood that the preset prompt text involved in this application includes all rules such as subject classification, label matching, format output, and invalid information filtering.

[0034] In one specific embodiment provided in this application, a preset prompt text is obtained, and the target image and the preset prompt text are input into an image-to-text conversion model. The image-to-text conversion model identifies the text content in the target image. Based on this, according to the content of the preset prompt text, the image-to-text conversion model is guided to extract at least one target subject from the identified text content, and based on each extracted target subject, the target subject text corresponding to each target subject is determined.

[0035] To facilitate understanding, this application first explains the construction method of the preset prompt text and the image-to-text conversion model in the following manner, and then further explains step 104.

[0036] In one specific embodiment provided in this application, the preset prompt text is determined in the following manner: Get at least one task prompt type; Based on each task prompt type, set at least one prompt text corresponding to each task prompt type; Based on each task type and at least one corresponding prompt text, construct preset prompt text.

[0037] The task prompt type can be understood as a classification dimension that modularly divides the preset prompt text based on the attribute features of the text contained in the image. For example, it could be a general task prompt type and a subject-specific task prompt type (such as a language arts task prompt type or a mathematics task prompt type).

[0038] The prompt text can be understood as a sub-text instruction module designed separately for a single task prompt type, which only contains the subject task recognition, classification, and output rules corresponding to that type.

[0039] In one specific embodiment provided in this application, at least one task prompt type is identified and determined based on the attribute tagging system of text in an image, serving as the classification basis for the modular construction of preset prompt text. The task prompt type can be a general task prompt type or a subject-specific task prompt type.

[0040] In one example, for each identified task prompt type, at least one prompt text is designed to match it. The prompt text design needs to include: content recognition rules for that type (e.g., filtering irrelevant information, defining the scope of the task content); subject attribute matching rules for that type (e.g., general task prompt types can match across disciplines, subject-specific task prompt types only match the corresponding subject); and output format rules for that type (e.g., tag quantity limits, serial number rules, subject-starting format, etc.). All the task prompt types and their corresponding prompt texts obtained above are integrated to obtain the preset prompt text.

[0041] According to a specific embodiment provided in this application, corresponding prompt text is set by different task prompt types, so that after the subsequent image-to-text conversion model recognizes the content in the image, it can accurately determine the corresponding prompt text for the content in the image.

[0042] Furthermore, in one specific embodiment provided in this application, based on each task prompt type, at least one prompt text corresponding to each task prompt type is set, including: Determine the initial task prompt type, wherein the initial task prompt type is any one of the various task prompt types; Based on the initial task prompt type, obtain at least one subject corresponding to the initial task prompt type, and determine the subject attribute information corresponding to each subject; Based on the subject attribute information corresponding to each subject, determine the subject extraction rules and subject text extraction rules for each subject. Based on the subject extraction rules and subject text extraction rules corresponding to each subject, at least one prompt text corresponding to the initial task prompt type is obtained.

[0043] The initial task prompt type can be understood as a single type sample arbitrarily selected from all task prompt types, which is the baseline type used by the image-to-text conversion model to complete the prompt text setting in advance.

[0044] Subjects can be understood as the learning subjects to which the text content contained in the image belongs, including conventional subjects such as Chinese, mathematics, English, and physics, as well as unknown subjects.

[0045] Subject attribute information can be understood as the unique textual attribute features and tag system corresponding to each subject, which is the core basis for distinguishing different subject types.

[0046] Subject extraction rules can be understood as rules formulated based on subject attribute information to determine whether text content belongs to that subject.

[0047] Subject text extraction rules can be understood as rules that accompany subject extraction rules and are used to extract specific assignment content under that subject and to perform attribute tagging and format standardization.

[0048] In one specific embodiment provided in this application, one type is arbitrarily selected as the initial task prompt type from all the task prompt types that have been identified (such as general task prompt types, language task prompt types, and mathematics task prompt types). For the selected initial task prompt type, at least one corresponding subject is first matched. Then, based on the subject and category association information, the subject attribute information corresponding to each matched subject is extracted and determined, including general attribute tags, subject-specific attribute tags, and applicable rules for the tags. Based on the subject attribute information obtained above, a two-layer extraction rule is formulated for each matched subject.

[0049] Among them, the formulation of subject extraction rules includes: clarifying the criteria for judging the homework content of the subject, the format of subject identification, and the rules for defining unknown subjects, so as to conform to the subject division requirements of the handover document.

[0050] Developing subject-specific text extraction rules includes: defining the scope of the subject's assignment content to be extracted, attribute tag matching rules (such as no more than 3 tags, sorted by relevance), and output format rules (such as Arabic numerals for assignments, Roman numerals for tags, and exclusive tags for textbook scenarios).

[0051] The aforementioned subject extraction rules and subject text extraction rules are transformed into natural language instructions that can be understood by the multimodal large model. After being arranged according to logical hierarchy, at least one prompt text corresponding to the initial task prompt type is generated. If the initial task prompt type corresponds to multiple subjects, an exclusive prompt text is generated for each subject. If it corresponds to only a single subject, an exclusive prompt text for that subject is generated.

[0052] In one specific embodiment provided in this application, the image-to-text conversion model is trained in the following manner: Obtain sample images, and obtain at least one sample subject corresponding to the sample images and sample subject text corresponding to each sample subject, wherein the sample images include subject tasks to be labeled; The sample image and the preset prompt text are input into the image-to-text conversion model to obtain at least one predicted subject and the predicted subject text corresponding to each predicted subject output by the image-to-text conversion model. The preset prompt text is used to guide the image-to-text conversion model to extract at least one subject and the subject text corresponding to each subject from the subject task to be labeled. Based on each predicted discipline, the predicted discipline text corresponding to each predicted discipline, each sample discipline, and the sample discipline text corresponding to each sample discipline, the model loss of the image-text conversion model is determined until the model training stops.

[0053] Among them, the image-to-text conversion model can be understood as a multimodal large model that can simultaneously process image and text dual-modal information and can directly extract the subject and the corresponding tagged text from the target image.

[0054] Sample images can be understood as images containing the subject-specific tasks to be labeled, used for model training. For example, they can be real job images in a small dataset used for supervised fine-tuning, which are image modality sample data for model learning.

[0055] The sample discipline can be understood as the real, well-annotated discipline type corresponding to the discipline task to be labeled in the sample image, which is the real label extracted by the model discipline.

[0056] Sample subject text can be understood as the labeled, real task text that corresponds one-to-one with each sample subject, containing specific task content and attribute tags. It is the real tag for model text extraction and labeling.

[0057] The subject-specific task to be labeled can be understood as the content information of the subject-specific task contained in the sample image that needs to be identified and classified by the model. For example, it could be the subject-specific learning task that a student or parent needs to complete, which is the core processing object of the model.

[0058] Subject prediction can be understood as the subject recognition result that the model autonomously outputs after inputting sample images and preset prompt text during the training process. It is the model's prediction value for the subject of the sample.

[0059] Predicted subject text can be understood as the structured text results that the model autonomously outputs for each predicted subject during the training process; it is the model's prediction value for the sample subject text.

[0060] Model loss can be understood as the quantitative difference between the predicted results output by the model and the actual labeled results, which is used to guide the updating of model parameters and thus improve the model's capabilities.

[0061] The model training termination condition can be understood as a pre-set critical condition for determining the completion of model training. It is usually the model loss falling below a preset threshold, the number of model iterations reaching a preset value, or the model prediction accuracy reaching a preset standard. Training can be terminated once any one of these conditions is met.

[0062] In one specific implementation provided in this application, a supervised dataset for model training is first constructed, and sample images are obtained. For example, this could be real homework images containing the subject to be labeled, with at least one sample subject (such as Chinese language and mathematics) labeled for each image. For each sample subject, corresponding sample subject text is labeled, including structured homework content and attribute tags. The sample images and preset prompt text are simultaneously input into an untrained image-to-text conversion model awaiting fine-tuning. The model performs forward inference based on bimodal input, autonomously completing subject extraction and text structuring, and finally outputting at least one predicted subject and corresponding predicted subject text. The predicted subject and predicted subject text output by the model are compared with the labeled sample subject and sample subject text. The quantitative difference between the two is calculated using a preset loss function (e.g., cross-entropy loss) to obtain the model loss. Based on the calculated model loss, the network parameters of the image-to-text conversion model are updated through backpropagation and gradient descent algorithms, allowing the model to learn in a direction where the predicted results are closer to the real labeled results. Repeat the above steps, continuously inputting different sample images and corresponding annotation information, calculating the model loss and updating the parameters, until the model loss meets the model training stopping condition (such as the loss value dropping below the threshold), then terminate the training and obtain the trained image-to-text conversion model.

[0063] Based on the preset prompt text and image-to-text conversion model obtained above in this application, this application adopts the following method to obtain at least one target subject output by the image-to-text conversion model and the subject text corresponding to each target subject.

[0064] In one specific embodiment provided in this application, obtaining at least one target subject and the corresponding subject text of each target subject output by the image-to-text conversion model includes: Identify the subject task to be labeled in the target image, and determine the target task type corresponding to the subject task to be labeled; Based on the target task type, determine the target prompt text corresponding to the target task type from the preset prompt text; Based on the target prompt text, at least one target subject and the corresponding subject text for each target subject are extracted from the subject task to be labeled.

[0065] The subject-specific task to be labeled can be understood as the text content information in the target image that needs to be recognized and classified by the model. For example, it could be a subject-specific learning task that a student or parent needs to complete. In this case, invalid information such as greetings and irrelevant prompts is excluded.

[0066] The target task type can be understood as the attribute tag type corresponding to the subject task to be labeled, which is the core attribute definition of the text content. For example, it can be understood as the homework content attribute tag, which includes general types (offline practice, error correction, reminders, etc.) and subject-specific types (mathematical calculation practice, Chinese vocabulary mastery, English word or sentence mastery, etc.).

[0067] The target prompt text can be understood as the prompt text that is selected from the preset prompt text and is completely compatible with the target task type. It is a subdivision rule of the preset prompt text, which includes the extraction, classification and output rules of the corresponding target task type.

[0068] In one specific embodiment provided in this application, the image-to-text conversion model performs bimodal fusion understanding on the input target image. First, it extracts all subject tasks to be labeled from the visual features of the target image (e.g., layout, text regions), removing irrelevant and invalid information (e.g., greetings, dates, signatures, etc., which are not part of the assignment). Based on this, according to a preset attribute labeling system, it performs attribute feature matching on the extracted subject tasks to determine the target task type corresponding to that subject task. The model uses the determined target task type as a filtering criterion, retrieving and extracting the prompt text that corresponds one-to-one with the target task type from the pre-input preset prompt text; this is the target prompt text. If the subject task to be labeled matches multiple target task types, it extracts all corresponding sub-prompt text modules and integrates them into the target prompt text. Furthermore, the image-to-text conversion model uses the target prompt text as a guiding rule to perform subject classification and structured text extraction on the identified subject tasks to be labeled. Specifically, based on the subject division rules in the target prompt text, at least one target subject (such as Chinese or mathematics) is extracted from the subject task to be labeled. Then, for each target subject, according to the output format requirements and attribute tag limitations in the target prompt text, the specific homework content is extracted and matched with the corresponding attribute tags, and finally, the structured subject text corresponding to each target subject is generated.

[0069] Specifically, based on the target prompt text, at least one target subject and the corresponding subject text for each target subject are extracted from the subject task to be labeled, including S1042-S1048: S1042. Based on the target prompt text, obtain at least one target subject type extraction rule and the target subject text extraction rule corresponding to each target subject type.

[0070] Among them, the target subject type extraction rule can be understood as the rule extracted from the target prompt text, which is used to filter and determine the subject to which the subject task to be labeled belongs.

[0071] The target subject text extraction rules can be understood as rules that correspond one-to-one with the extraction rules for each target subject type, used to extract specific text content under that subject and to perform attribute tagging.

[0072] S1044. Based on the extraction rules for each target subject type, at least one target subject is extracted from the subject task to be labeled.

[0073] S1046. In the target image, determine the text location information corresponding to each target subject.

[0074] The text location information can be understood as the visual location and regional features of the assignment content corresponding to each target subject in the target image. For example, it can be non-textual visual features such as the layout of the assignment image, the distribution of text regions, and the location of handwritten marks.

[0075] In one specific embodiment provided in this application, determining the text location information corresponding to each target subject in the target image includes: Determine the initial target discipline, wherein the initial target discipline is any one of the target disciplines; Determine the subject semantics of the initial target subject, and determine the task semantics corresponding to the subject task to be labeled; Based on the semantics of the task to be labeled and the semantics of the discipline, the discipline text corresponding to the initial target discipline is obtained by filtering. Determine the location information of the subject text corresponding to the initial target subject, and obtain the text location information corresponding to the initial target subject.

[0076] The initial target subject can be understood as a single subject sample arbitrarily selected from all extracted target subjects. It is the benchmark subject used by the image-to-text conversion model to first determine the text location information, and then the determination method is applied to all other target subjects.

[0077] Subject semantics can be understood as the inherent semantic features and core attributes of the initial target subject. For example, the semantics of the Chinese language subject can be "language learning tasks related to words, reading aloud, writing, and texts", while the semantics of the mathematics subject can be "mathematical learning tasks related to calculation, formulas, and exercises".

[0078] The semantics of the task to be labeled can be understood as the overall semantic features of the subject task to be labeled in the target image. It is the overall semantic expression of the task content obtained by the model after understanding through multimodal fusion. For example, the overall semantics of "copying words and dictating texts" is "the task of mastering basic vocabulary and texts in the English subject".

[0079] In one specific embodiment provided in this application, one subject is arbitrarily selected as the initial target subject from all target subjects (such as Chinese, mathematics, and English) extracted by the image-to-text conversion model. Based on its cross-modal semantic understanding capabilities, the image-to-text conversion model extracts the subject semantics of the initial target subject and performs overall semantic analysis on the subject task to be labeled in the target image, extracting the semantics of the task to be labeled. On this basis, the image-to-text conversion model performs semantic matching between the semantics of the task to be labeled and the subject semantics of the initial target subject, selecting text content that matches the semantics of the initial target subject from all subject tasks to be labeled in the target image; this content is the subject text corresponding to the initial target subject. Based on the selected subject text corresponding to the initial target subject, the image-to-text conversion model performs visual feature localization in the target image, extracting the specific visual position and regional features of the subject text in the image (such as layout lines, text blocks, handwritten annotation areas, etc.); these features are the text position information corresponding to the initial target subject.

[0080] S1048. Based on the target subject text extraction rules corresponding to each target subject type and the text location information corresponding to each target subject, extract the subject text corresponding to each target subject from the subject task to be labeled.

[0081] In one specific embodiment provided in this application, the image-to-text conversion model performs structured rule parsing on the matched target prompt text, extracts at least one target subject type extraction rule from the target prompt text, and a target subject text extraction rule corresponding to each target subject type extraction rule.

[0082] Building upon the above, the image-to-text conversion model uses the target subject type extraction rules obtained from the analysis to perform subject matching and filtering on the identified tasks to be labeled. For example, it might compare the semantic and content features of the task to be labeled with the extraction rules for each subject type; a successful match is the target subject. It's important to understand that for task content that cannot be matched with any subject type extraction rule, it is extracted as an unknown subject task according to the rules, ultimately resulting in at least one target subject.

[0083] Furthermore, the image-to-text conversion model combines visual feature analysis of the target image. For each extracted target subject, it locates the visual region and position of the corresponding assignment content within the target image, determining text location information such as layout lines, text blocks, and handwritten annotation areas, thus binding the target subject with the image location. For each target subject, the model combines its corresponding text extraction rules with text location information to extract the specific assignment content within a specified area of ​​the target image, and performs attribute tagging and structuring according to the text extraction rules. For example, assignment items are numbered using Arabic numerals; no more than three attribute tags are matched to each assignment item; attribute tags are numbered using Roman numerals; and if textbook or teaching material scenarios are involved, additional textbook scenario tags are forcibly added. This generates standardized subject text for each target subject.

[0084] In the data processing methods described above in this application, this application may also employ the following method to determine at least one target subject corresponding to the subject task to be labeled in the target image and the subject text corresponding to each target subject. Specifically: Determine the target task type for the subject-specific task to be labeled in the target image; Based on the target task type, determine the target prompt text corresponding to the target image; The target image and the target prompt text are input into the image-to-text conversion model to obtain at least one target subject and the corresponding subject text output by the image-to-text conversion model.

[0085] The target task type can be understood as the attribute tag type corresponding to the subject task to be labeled, which is the core attribute definition of the text content. For example, it can be understood as the homework content attribute tag, which includes general types (offline practice, error correction, reminders, etc.) and subject-specific types (mathematical calculation practice, Chinese vocabulary mastery, English word or sentence mastery, etc.).

[0086] The target prompt text can be understood as the prompt text that is selected from the preset prompt text and is completely compatible with the target task type. It is a subdivision rule of the preset prompt text, which includes the extraction, classification and output rules of the corresponding target task type.

[0087] In one specific embodiment provided in this application, when the target task type corresponding to the subject task to be labeled in the target image is known, this application can also directly determine the target prompt text corresponding to the target task type from the preset prompt text. In this case, the target image and the target prompt text are input together into the image-to-text conversion model, eliminating the need for the image-to-text conversion model to identify the subject task to be labeled in the target image again, further improving the processing efficiency of the image-to-text model.

[0088] According to the data processing method described above in this application, by inputting the target image and the preset prompt text into the image-to-text conversion model, it is not necessary to use two models to perform text recognition on the target image and then further annotate it based on the recognition results, thereby effectively avoiding the generation of cascading errors caused by model collaboration.

[0089] The following is in conjunction with the appendix Figure 2 Taking the application of the data processing method provided in this application in subject content annotation as an example, the data processing method will be further explained. Figure 2 This application provides a flowchart illustrating a data processing method for subject content annotation according to an embodiment of the present application, which specifically includes the following steps: Step 202: Construct a sample dataset.

[0090] The sample dataset consists of multiple sample pairs. Each sample pair includes two parts: input and output. The input is a real image of the assignment. The output is a text string with a uniform format that completely corresponds to the content of the input image, containing the subject category and subject text from the assignment image.

[0091] Furthermore, by restricting the output rules of the strings, subsequent post-processing according to the rules is facilitated. For example, the current rule is that strings begin with "xx subject" to distinguish the overall subject, such as "[Chinese Language] Homework 1\n Homework 2\n [Mathematics] Homework", where "\n" is a newline character; the homework includes multiple specific text contents and corresponding category tags. For example, for the two text contents of the Chinese Language subject, it is constructed as "1. Homework Content 1\n I. Attribute Tag 1 II.\n Attribute Tag 2\n 2. Homework Content 2\n I. Attribute Tag 1". This output uses Arabic numerals to distinguish homework contents and Roman numerals to distinguish attribute tags, which facilitates subsequent secondary processing and display.

[0092] Step 204: Design preset prompt text that takes into account both recognition and classification.

[0093] The preset prompt text is used as a text modality input into the image-to-text conversion model. This allows the model to understand the image and identify and classify the subject matter to be labeled within it.

[0094] For example, the preset prompt text could be: "You are an image-based homework recognition and classification tool. Your core task is to identify from images whether they are homework content, pre-study, review, or other tasks that students and parents need to complete, and then break them down and classify them, outputting each subject's homework separately. Please strictly follow these rules:\n1. Output homework for each subject separately. If the subject of some homework is uncertain, please output these uncertain subject homework as 'Unknown Subject Homework'. Each subject's homework must begin with 'Chinese Homework:', 'Mathematics Homework:', 'English Homework:', 'Physics Homework:', 'Chemistry Homework:', 'Biology Homework:', 'Politics Homework:', 'Moral Education and Law Homework:', 'History Homework:', 'Geography Homework:', 'Science Homework:', 'Unknown Subject Homework:'.\n2. If some input text is prompts or greetings, please ignore these and do not output them.\n3. The following are the instructions for each subject..." The available categories for each subject, besides general categories, are as follows: Assignments for each subject can only be categorized under that subject's category. Assignments from any subject may be categorized under general categories. The subject-category association information is as follows: a. General: Offline practice, offline tasks, error correction, reminders, preview, review, questions, research, copying, memorization, handwritten posters, other textbook scenarios. b. Chinese: Vocabulary mastery, memorizing texts, reading aloud, dictation, composition writing, speaking practice, extracurricular reading. c. Mathematics: Calculation practice. d. English: Vocabulary / sentence mastery, memorizing texts, reading aloud, composition writing. 4. Each assignment should be categorized with no more than three categories, sorted from highest to lowest relevance to the assignment, using Roman numerals for numbering (e.g., I, II), and displayed on a new line after the assignment content. 5. Any scenario that might involve textbooks or teaching materials needs to be categorized under "Other Textbook Scenarios"!!! Step 206: Fine-tune the image-to-text conversion model.

[0095] Specifically, fine-tuning is performed based on a sample dataset not covered by Section 202. The training loss function typically calculates the difference between the model's predicted output and the sample data (e.g., cross-entropy loss). Through backpropagation and gradient descent, the model parameters are updated, gradually teaching it how to generate accurate and compliant job content, job category, and job attribute information based on any input job image.

[0096] Step 208: Image-to-text conversion model reasoning and result application.

[0097] After the image-to-text conversion model is trained, images are used as input in the image modality, and the pre-designed preset prompt text is input into the model in text form. This results in a subject-specific display of the assignment content and a tagged display of the specific content of the corresponding subject text. Subsequently, assignments can be displayed and managed at the subject level and attribute level by using Roman numerals and Arabic numerals for differentiation.

[0098] Corresponding to the above method embodiments, this application also provides data processing apparatus embodiments. Figure 3 A schematic diagram of the structure of a data processing apparatus according to an embodiment of this application is shown. Figure 3 As shown, the device includes: The acquisition unit 302 is configured to acquire a target image, wherein the target image includes a subject task to be labeled; The processing unit 304 is configured to input the target image and preset prompt text into the image-to-text conversion model to obtain at least one target subject and the subject text corresponding to each target subject output by the image-to-text conversion model. The preset prompt text is used to guide the image-to-text conversion model to extract at least one target subject and the subject text corresponding to each target subject from the subject task to be labeled.

[0099] Furthermore, the processing unit 304 is further configured as follows: Identify the subject task to be labeled in the target image, and determine the target task type corresponding to the subject task to be labeled; Based on the target task type, determine the target prompt text corresponding to the target task type from the preset prompt text; Based on the target prompt text, at least one target subject and the corresponding subject text for each target subject are extracted from the subject task to be labeled.

[0100] Furthermore, the processing unit 304 is further configured as follows: Based on the target prompt text, at least one target subject type extraction rule and the target subject text extraction rule corresponding to each target subject type are obtained; Based on the extraction rules for each target subject type, at least one target subject is extracted from the subject task to be labeled; In the target image, determine the text location information corresponding to each target subject; Based on the target subject text extraction rules corresponding to each target subject type and the text location information corresponding to each target subject, the subject text corresponding to each target subject is extracted from the subject task to be labeled.

[0101] Furthermore, the processing unit 304 is further configured as follows: Determine the initial target discipline, wherein the initial target discipline is any one of the target disciplines; Determine the subject semantics of the initial target subject, and determine the task semantics corresponding to the subject task to be labeled; Based on the semantics of the task to be labeled and the semantics of the discipline, the discipline text corresponding to the initial target discipline is obtained by filtering. Determine the location information of the subject text corresponding to the initial target subject, and obtain the text location information corresponding to the initial target subject.

[0102] Furthermore, the processing unit 304 is further configured as follows: Get at least one task prompt type; Based on each task prompt type, set at least one prompt text corresponding to each task prompt type; Based on each task type and at least one corresponding prompt text, construct preset prompt text.

[0103] Furthermore, the processing unit 304 is further configured as follows: Determine the initial task prompt type, wherein the initial task prompt type is any one of the various task prompt types; Based on the initial task prompt type, obtain at least one subject corresponding to the initial task prompt type, and determine the subject attribute information corresponding to each subject; Based on the subject attribute information corresponding to each subject, determine the subject extraction rules and subject text extraction rules for each subject. Based on the subject extraction rules and subject text extraction rules corresponding to each subject, at least one prompt text corresponding to the initial task prompt type is obtained.

[0104] Furthermore, the acquisition unit 302 is further configured as follows: Acquire an initial image and determine the sharpness assessment result of the initial image, wherein the initial image includes an initial subject task to be labeled, and the sharpness assessment result is determined based on the attribute parameters of the initial subject task to be labeled; If the sharpness assessment result meets the sharpness threshold, the initial image is determined as the target image; If the sharpness assessment result does not meet the sharpness threshold, the attribute parameters of the initial subject task to be labeled are adjusted until the sharpness assessment result of the initial image meets the sharpness threshold, thus obtaining the target image.

[0105] Furthermore, the processing unit 304 is also configured as follows: Determine the target task type for the subject-specific task to be labeled in the target image; Based on the target task type, determine the target prompt text corresponding to the target image; The target image and the target prompt text are input into the image-to-text conversion model to obtain at least one target subject and the corresponding subject text output by the image-to-text conversion model.

[0106] Furthermore, the image-to-text conversion model is trained in the following manner: Obtain sample images, and obtain at least one sample subject corresponding to the sample images and sample subject text corresponding to each sample subject, wherein the sample images include subject tasks to be labeled; The sample image and the preset prompt text are input into the image-to-text conversion model to obtain at least one predicted subject and the predicted subject text corresponding to each predicted subject output by the image-to-text conversion model. The preset prompt text is used to guide the image-to-text conversion model to extract at least one subject and the subject text corresponding to each subject from the subject task to be labeled. Based on each predicted discipline, the predicted discipline text corresponding to each predicted discipline, each sample discipline, and the sample discipline text corresponding to each sample discipline, the model loss of the image-text conversion model is determined until the model training stops.

[0107] The above is an illustrative scheme of a data processing apparatus according to this embodiment. It should be noted that the technical solution of this data processing apparatus and the technical solution of the data processing method described above belong to the same concept. For details not described in detail in the technical solution of the data processing apparatus, please refer to the description of the technical solution of the data processing method described above.

[0108] Figure 4 A structural block diagram of a computing device according to an embodiment of this application is shown. The components of the computing device 400 include, but are not limited to, a memory 410 and a processor 420. The processor 420 is connected to the memory 410 via a bus 430, and a database 450 is used to store data.

[0109] The computing device 400 also includes an access device 440, which enables the computing device 400 to communicate via one or more networks 460. Examples of these networks include Public Switched Telephone Network (PSTN), Local Area Network (LAN), Wide Area Network (WAN), Personal Area Network (PAN), or combinations of communication networks such as the Internet. The access device 440 may include one or more of any type of wired or wireless network interface (e.g., a network interface card (NIC)), such as an IEEE 802.11 Wireless Local Area Network (WLAN) wireless interface, a Wi-MAX (Worldwide Interoperability for Microwave Access) interface, an Ethernet interface, a Universal Serial Bus (USB) interface, a cellular network interface, a Bluetooth interface, a Near Field Communication (NFC) interface, and so on.

[0110] In one embodiment of this application, the aforementioned components of the computing device 400 and Figure 4 Other components, not shown, can also be connected to each other, for example, via a bus. It should be understood that... Figure 4 The block diagram of the computing device shown is for illustrative purposes only and is not intended to limit the scope of this application. Those skilled in the art can add or replace other components as needed.

[0111] The computing device 400 can be any type of stationary or mobile computing device, including mobile computers or mobile computing devices (e.g., tablet computers, personal digital assistants, laptop computers, notebook computers, netbooks, etc.), mobile phones (e.g., smartphones), wearable computing devices (e.g., smartwatches, smart glasses, etc.) or other types of mobile devices, or stationary computing devices such as desktop computers or personal computers (PCs). The computing device 400 can also be a mobile or stationary server.

[0112] The processor 420 is used to execute the following computer program / instructions, which, when executed by the processor, implement the steps of the above-described data processing method.

[0113] The above is an illustrative scheme of a computing device according to this embodiment. It should be noted that the technical solution of this computing device and the technical solution of the data processing method described above belong to the same concept. For details not described in detail in the technical solution of the computing device, please refer to the description of the technical solution of the data processing method described above.

[0114] An embodiment of this specification also provides a computer-readable storage medium storing a computer program / instructions that, when executed by a processor, implement the steps of the above-described data processing method.

[0115] The above is an illustrative scheme of a computer-readable storage medium according to this embodiment. It should be noted that the technical solution of this storage medium and the technical solution of the data processing method described above belong to the same concept. For details not described in detail in the technical solution of the storage medium, please refer to the description of the technical solution of the data processing method described above.

[0116] An embodiment of this specification also provides a computer program product, including a computer program / instructions that, when executed by a processor, implement the steps of the above-described data processing method.

[0117] The above is an illustrative scheme of a computer program product according to this embodiment. It should be noted that the technical solution of this computer program product and the technical solution of the data processing method described above belong to the same concept. For details not described in detail in the technical solution of the computer program product, please refer to the description of the technical solution of the data processing method described above.

[0118] The foregoing has described specific embodiments of this application. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims may be performed in a different order than that shown in the embodiments and may still achieve the desired results. Furthermore, the processes depicted in the drawings do not necessarily require the specific or sequential order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0119] The computer instructions include computer program code, which may be in the form of source code, object code, executable file, or certain intermediate forms. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording media, USB flash drive, portable hard drive, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc. It should be noted that the content included in the computer-readable medium may be appropriately added or removed according to the requirements of patent practice. For example, in some regions, according to patent practice, computer-readable media may not include electrical carrier signals and telecommunication signals.

[0120] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that this application is not limited to the described order of actions, as some steps may be performed in other orders or simultaneously according to this application. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are preferred embodiments, and the actions and modules involved are not necessarily essential to this application.

[0121] In the above embodiments, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0122] The preferred embodiments disclosed above are merely illustrative of this application. The optional embodiments do not exhaustively describe all details, nor do they limit the invention to the specific implementations described. Clearly, many modifications and variations can be made based on the content of this application. These embodiments are selected and specifically described in this application to better explain the principles and practical applications of this application, thereby enabling those skilled in the art to better understand and utilize this application. This application is limited only by the claims and their full scope and equivalents.

Claims

1. A data processing method, characterized in that, include: Acquire a target image, wherein the target image includes a subject task to be labeled; The target image and the preset prompt text are input into the image-to-text conversion model to obtain at least one target subject and the corresponding subject text of each target subject output by the image-to-text conversion model. The preset prompt text is used to guide the image-to-text conversion model to extract at least one target subject and the corresponding subject text of each target subject from the subject task to be labeled.

2. The method as described in claim 1, characterized in that, Obtaining at least one target subject and the corresponding subject text from the image-to-text conversion model includes: Identify the subject task to be labeled in the target image, and determine the target task type corresponding to the subject task to be labeled; Based on the target task type, determine the target prompt text corresponding to the target task type from the preset prompt text; Based on the target prompt text, at least one target subject and the corresponding subject text for each target subject are extracted from the subject task to be labeled.

3. The method as described in claim 2, characterized in that, Based on the target prompt text, at least one target subject and the corresponding subject text for each target subject are extracted from the subject task to be labeled, including: Based on the target prompt text, at least one target subject type extraction rule and the target subject text extraction rule corresponding to each target subject type are obtained; Based on the extraction rules for each target subject type, at least one target subject is extracted from the subject task to be labeled; In the target image, determine the text location information corresponding to each target subject; Based on the target subject text extraction rules corresponding to each target subject type and the text location information corresponding to each target subject, the subject text corresponding to each target subject is extracted from the subject task to be labeled.

4. The method as described in claim 3, characterized in that, In the target image, the text location information corresponding to each target subject is determined, including: Determine the initial target discipline, wherein the initial target discipline is any one of the target disciplines; Determine the subject semantics of the initial target subject, and determine the task semantics corresponding to the subject task to be labeled; Based on the semantics of the task to be labeled and the semantics of the discipline, the discipline text corresponding to the initial target discipline is obtained by filtering. Determine the location information of the subject text corresponding to the initial target subject, and obtain the text location information corresponding to the initial target subject.

5. The method according to any one of claims 1 to 4, characterized in that, The preset prompt text is determined in the following manner: Get at least one task prompt type; Based on each task prompt type, set at least one prompt text corresponding to each task prompt type; Based on each task type and at least one corresponding prompt text, construct preset prompt text.

6. The method as described in claim 5, characterized in that, Based on each task prompt type, set at least one prompt text corresponding to each task prompt type, including: Determine the initial task prompt type, wherein the initial task prompt type is any one of the various task prompt types; Based on the initial task prompt type, obtain at least one subject corresponding to the initial task prompt type, and determine the subject attribute information corresponding to each subject; Based on the subject attribute information corresponding to each subject, determine the subject extraction rules and subject text extraction rules for each subject. Based on the subject extraction rules and subject text extraction rules corresponding to each subject, at least one prompt text corresponding to the initial task prompt type is obtained.

7. The method as described in claim 1, characterized in that, Acquire the target image, including: Acquire an initial image and determine the sharpness assessment result of the initial image, wherein the initial image includes an initial subject task to be labeled, and the sharpness assessment result is determined based on the attribute parameters of the initial subject task to be labeled; If the sharpness assessment result meets the sharpness threshold, the initial image is determined as the target image; If the sharpness assessment result does not meet the sharpness threshold, the attribute parameters of the initial subject task to be labeled are adjusted until the sharpness assessment result of the initial image meets the sharpness threshold, thus obtaining the target image.

8. The method as described in claim 1, characterized in that, The method further includes: Determine the target task type for the subject-specific task to be labeled in the target image; Based on the target task type, determine the target prompt text corresponding to the target image; The target image and the target prompt text are input into the image-to-text conversion model to obtain at least one target subject and the corresponding subject text output by the image-to-text conversion model.

9. The method as described in claim 1, characterized in that, The image-to-text conversion model is trained in the following manner: Obtain sample images, and obtain at least one sample subject corresponding to the sample images and sample subject text corresponding to each sample subject, wherein the sample images include subject tasks to be labeled; The sample image and the preset prompt text are input into the image-to-text conversion model to obtain at least one predicted subject and the predicted subject text corresponding to each predicted subject output by the image-to-text conversion model. The preset prompt text is used to guide the image-to-text conversion model to extract at least one subject and the subject text corresponding to each subject from the subject task to be labeled. Based on each predicted discipline, the predicted discipline text corresponding to each predicted discipline, each sample discipline, and the sample discipline text corresponding to each sample discipline, the model loss of the image-text conversion model is determined until the model training stops.

10. A data processing apparatus, characterized in that, include: An acquisition unit is configured to acquire a target image, wherein the target image includes a subject task to be labeled; The processing unit is configured to input the target image and preset prompt text into an image-to-text conversion model to obtain at least one target subject and the subject text corresponding to each target subject output by the image-to-text conversion model. The preset prompt text is used to guide the image-to-text conversion model to extract at least one target subject and the subject text corresponding to each target subject from the subject task to be labeled.

11. A computing device, characterized in that, include: Memory and processor; The memory is used to store computer programs / instructions, and the processor is used to execute the computer programs / instructions, which, when executed by the processor, implement the steps of the method according to any one of claims 1 to 9.

12. A computer-readable storage medium storing a computer program / instructions, characterized in that, When the computer program / instructions are executed by the processor, they implement the steps of the method according to any one of claims 1 to 9.

13. A computer program product comprising a computer program / instructions, characterized in that, When the computer program / instructions are executed by the processor, they implement the steps of the method according to any one of claims 1 to 9.

Images