Data processing method and device

Abstract

The embodiment of the invention provides a data processing method and device, and the method comprises the steps: inputting a target image containing at least one to-be-detected object into a detection model, and obtaining a detection frame of each to-be-detected object; according to the detection box of each to-be-detected object, a reference result of each to-be-detected object and at least one candidate judgment type, a prompt text is constructed, the reference result is used for judging the type of the to-be-detected object, and the candidate judgment type is represented by a basic semantic unit of a multi-modal data processing model; the target image and the prompt text are input into a multi-modal data processing model, a type sequence containing at least one target judgment type is obtained, and the target judgment type is determined from at least one candidate judgment type; matching at least one target judgment type in the type sequence with at least one to-be-detected object to obtain a target judgment type corresponding to each to-be-detected object; in a complex multi-frame scene, the dislocation rate is obviously reduced, and the real-time performance and reliability are improved.

Description

Technical Field

[0001] This specification relates to the field of computer technology, and in particular to a data processing method. One or more embodiments of this specification also relate to a data processing apparatus, a computing device, a computer-readable storage medium, and a computer program product. Background Technology

[0002] In the field of education, automated homework grading technology aims to use artificial intelligence methods such as computer vision and natural language processing to identify, understand and judge students' answers, thereby replacing or assisting manual grading.

[0003] However, the grading output information based on large models is usually large and time-consuming, making it difficult to achieve low-latency grading. In scenarios with multiple answer boxes, the results and positions are easily misaligned, leading to a decrease in grading accuracy. Summary of the Invention

[0004] In view of the above, embodiments of this specification provide a data processing method. One or more embodiments of this specification also relate to a data processing apparatus, a computing device, a computer-readable storage medium, and a computer program product, to address the technical deficiencies existing in the prior art.

[0005] According to a first aspect of the embodiments of this specification, a data processing method is provided, comprising: Input the target image containing at least one object to be detected into the detection model to obtain the detection bounding box for each object to be detected. Based on the detection bounding box of each object to be detected, the reference result of each object to be detected, and at least one candidate determination type, a prompt text is constructed, wherein the reference result is used to determine the type of the object to be detected, and the candidate determination type is represented by the basic semantic unit of the multimodal data processing model. The target image and the prompt text are input into a multimodal data processing model to obtain a type sequence containing at least one target determination type, wherein the target determination type is determined from the at least one candidate determination type; At least one target determination type in the type sequence is matched with at least one object to be detected to obtain the target determination type corresponding to each object to be detected.

[0006] According to a second aspect of the embodiments of this specification, a data processing apparatus is provided, comprising: The detection module is configured to input a target image containing at least one object to be detected into the detection model to obtain detection bounding boxes for each object to be detected. The construction module is configured to construct prompt text based on the detection box of each object to be detected, the reference result of each object to be detected, and at least one candidate determination type, wherein the reference result is used to determine the type of the object to be detected, and the candidate determination type is represented by the basic semantic unit of the multimodal data processing model. The acquisition module is configured to input the target image and the prompt text into a multimodal data processing model to obtain a type sequence containing at least one target determination type, wherein the target determination type is determined from the at least one candidate determination type; The matching module is configured to match at least one target determination type in the type sequence with the at least one object to be detected, thereby obtaining the target determination type corresponding to each object to be detected.

[0007] According to a third aspect of the embodiments of this specification, a computing device is provided, comprising: Memory and processor; The memory is used to store computer programs / instructions, and the processor is used to execute the computer programs / instructions, which, when executed by the processor, implement the steps of the above-described data processing method.

[0008] According to a fourth aspect of the embodiments of this specification, a computer-readable storage medium is provided that stores a computer program / instructions that, when executed by a processor, implement the steps of the data processing method described above.

[0009] According to a fifth aspect of the embodiments of this specification, a computer program product is provided, including a computer program / instructions that, when executed by a processor, implement the steps of the above-described data processing method.

[0010] This specification provides a data processing method in one embodiment that utilizes a detection model to quickly locate each object to be detected in a target image, obtaining the detection bounding box for each object, providing accurate input for subsequent analysis. Based on the detection bounding box, reference results, and candidate decision types represented using basic language units, a prompt text is constructed. The target image and the prompt text are input into a multimodal data processing model. Leveraging the powerful contextual understanding capabilities of the multimodal data processing model, a type sequence containing at least one target decision type is obtained, which is determined from at least one candidate decision type. By strictly matching the type sequence with the objects to be detected, it is ensured that each object to be detected can obtain a corresponding target decision type, thereby significantly reducing the misalignment rate in complex multi-bounding scenarios and improving the real-time performance and reliability of the grading system. Attached Figure Description

[0011] Figure 1This is a schematic diagram illustrating a data processing method provided in one embodiment of this specification. Figure 2 This is a flowchart illustrating a data processing method provided in one embodiment of this specification; Figure 3 This is a flowchart illustrating the processing procedure of a data processing method provided in one embodiment of this specification. Figure 4 This is a schematic diagram illustrating image processing in a data processing method provided in one embodiment of this specification; Figure 5 This is a schematic diagram of the structure of a data processing apparatus provided in one embodiment of this specification; Figure 6 This is a structural block diagram of a computing device provided in one embodiment of this specification. Detailed Implementation

[0012] Many specific details are set forth in the following description to provide a full understanding of this specification. However, this specification can be implemented in many other ways than those described herein, and those skilled in the art can make similar extensions without departing from the spirit of this specification. Therefore, this specification is not limited to the specific implementations disclosed below.

[0013] The terminology used in one or more embodiments of this specification is for the purpose of describing particular embodiments only and is not intended to be limiting of the one or more embodiments of this specification. The singular forms “a,” “described,” and “the” as used in one or more embodiments of this specification and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used in one or more embodiments of this specification refers to and includes any or all possible combinations of one or more associated listed items.

[0014] It should be understood that although the terms first, second, etc., may be used to describe various information in one or more embodiments of this specification, such information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, first may also be referred to as second without departing from the scope of one or more embodiments of this specification, and similarly, second may also be referred to as first. Depending on the context, the word "if" as used herein may be interpreted as "when," "when," or "in response to a determination."

[0015] Furthermore, it should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in one or more embodiments of this specification are all information and data authorized by the user or fully authorized by all parties. Moreover, the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and corresponding operation entry points are provided for users to choose to authorize or refuse.

[0016] First, the terms and concepts used in one or more embodiments of this specification will be explained.

[0017] Multi-modal Large Language Model: A large-scale pre-trained language model that can simultaneously process text and visual input and perform cross-modal reasoning and generation. It can be simply described as a multi-modal large model.

[0018] Large Language Model (LLM) is a natural language processing model based on deep learning. It is pre-trained on massive amounts of text data and has the ability to understand and generate language, and can handle a variety of natural language tasks.

[0019] SFT: Supervised Fine-Tuning, is a method of supervised training based on an existing pre-trained model using manually labeled data, to adapt the model to a specific task.

[0020] Prompt: This is a prompt text used to guide the large language model in generating the task description text according to the specified goal. The prompt text in the embodiments of this specification uses a structured template design.

[0021] Token: The smallest unit of text processing by the model, which can be a character, a word, or a subword. In the embodiments of this specification, the token design is optimized in the output stage, so that each grading result can be represented by a single token, thereby reducing inference time.

[0022] OCR: Optical Character Recognition, is a technology that converts text in an image into editable text.

[0023] Object detection is a computer vision task designed to identify the location and category of target objects in an image or video, typically outputting a bounding box with category labels. In the embodiments described in this specification, it is used to detect the answer area in an assignment image.

[0024] In the education field, automated homework grading technology aims to use artificial intelligence methods such as computer vision and natural language processing to identify, understand, and judge student answers, thereby replacing or assisting manual grading. Such systems need to handle a variety of question types, from objective questions (such as multiple choice, true / false, and fill-in-the-blank) to semi-subjective questions (such as mental arithmetic, vertical calculation, and dictation) and even subjective questions (such as short answer, drawing, and open-ended questions). Because different question types differ significantly in answer format, answer uniqueness, and scoring criteria, building a universal, efficient, and accurate automated grading system that can handle all question types has always been a major challenge for the industry.

[0025] Currently, mainstream automated grading solutions in the industry can be mainly divided into three categories. The first category is the traditional process based on object detection, OCR handwriting recognition, and rule comparison. This method first locates the answer area through object detection, then uses OCR technology to recognize the handwritten content, and finally compares the answers according to preset rules. This solution is usually designed with specific question types (such as mental arithmetic, vertical calculation, dictation, multiple choice questions, and mathematical word problems), and lacks versatility. The second category is grading based on large language models for pure text input. This solution relies entirely on OCR to convert the entire page or part of the handwritten answer into text, and then uses a large language model for content understanding and evaluation. Its effectiveness is limited by the recognition accuracy of OCR. The third category is grading based on publicly available multimodal large language models. This type of method can directly receive image input, avoiding errors in the OCR process, and has the basic ability to handle mixed text and image content. However, the output of multimodal large language models is natural language text, which has uncontrollable length, high latency, and cannot accurately mark the location of each answer.

[0026] Despite the progress made by the aforementioned technologies, they all have significant limitations. First, they lack versatility. Traditional solutions require customized detection, recognition, and rule matching modules for different question types, resulting in cumbersome processes and an inability to uniformly handle complex question types such as drawing questions and open-ended questions. For these complex question types, OCR and rule matching methods are difficult to effectively utilize, leading to a decrease in grading accuracy when recognition rates are low. Second, they lack real-time performance. Solutions based on large models often output natural language of varying lengths, resulting in high inference latency and hindering low-latency real-time grading and feedback. Third, the output results are uncontrollable. The natural language text generated by large models contains a lot of redundant information and has unstable formats, requiring complex post-processing parsing. This is especially problematic when dealing with multiple answer areas, easily leading to misalignment between the grading conclusion and the specific answer location. Furthermore, they underutilize location information. Some solutions only output the grading conclusion without returning the specific answer area location, and most solutions fail to fully utilize key metadata such as answer type (e.g., judgment, fill-in-the-blank, drawing) and the location information of the specific answer area to assist in evaluation, limiting the improvement of grading accuracy.

[0027] The data processing method provided in the embodiments of this specification utilizes the image and semantic understanding capabilities of a multimodal large model, combined with efficient answer box detection and task format design, to achieve a highly versatile, real-time, and accurate homework grading method.

[0028] This specification provides a data processing method, and also relates to a data processing apparatus, a computing device, a computer-readable storage medium, and a computer program product, which will be described in detail in the following embodiments.

[0029] See Figure 1 , Figure 1 A schematic diagram of a data processing method according to an embodiment of this specification is shown.

[0030] Specifically, the data processing method is applied to a data processing system, which includes an end-side device 102 and a server 104. The end-side device 102 is used to send a target image and a reference result of the object to be detected in the target image to the server 104.

[0031] A detection model and a multimodal data processing model are trained in server 104. When server 104 receives a target image and a reference result of the object to be detected in the target image sent by terminal device 102, the target image containing at least one object to be detected is input into the detection model to obtain detection boxes for each object to be detected. Based on the detection boxes of each object to be detected, the reference results of each object to be detected, and at least one candidate determination type, a prompt text is constructed. The reference result is used to determine the type of the object to be detected, and the candidate determination type is represented using the basic semantic unit of the multimodal data processing model. The target image and the prompt text are input into the multimodal data processing model to obtain a type sequence containing at least one target determination type. The target determination type is determined from the at least one candidate determination type. At least one target determination type in the type sequence is matched with the at least one object to be detected to obtain the target determination type corresponding to each object to be detected. The target determination type corresponding to each object to be detected is returned to terminal device 102.

[0032] The edge device 102 may include a browser, an app (application), or a web application such as an H5 (Hypertext Markup Language 5) application, a lightweight application (also known as a mini-program), or a cloud application. The edge device can be developed based on a software development kit (SDK) provided by the server, such as a real-time communication (RTC) SDK. The edge device can be deployed in an electronic device and depends on the device's operation or certain apps within the device to run. The electronic device may have a display screen and support information browsing, such as a personal mobile terminal like a mobile phone, tablet, or personal computer. Various other types of applications can also be configured in the electronic device, such as human-computer interaction applications, model training applications, data processing applications, web browser applications, shopping applications, search applications, instant messaging tools, email clients, and social media platform software.

[0033] Server 104 can be understood as a server providing various services, including physical servers and cloud servers. Examples include servers providing communication services to multiple clients, servers supporting backend training of models used on clients, and servers processing data sent by clients. It's important to note that Server 104 can be implemented as a distributed server cluster composed of multiple servers, or as a single server. Server 104 can also be a server in a distributed system, or a server integrated with blockchain. Server 104 can also be a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms, or an intelligent cloud computing server or intelligent cloud host with artificial intelligence technology.

[0034] It is worth noting that the data processing method provided in the embodiments of this specification can be executed by the server 104. In other embodiments of this specification, the detection model and / or multimodal data processing model can be deployed in the edge device 102, so that the edge device 102 can also have similar functions to the server 104, thereby executing the data processing method provided in the embodiments of this specification. In other embodiments, the data processing method provided in the embodiments of this specification can also be jointly executed by the edge device 102 and the server 104.

[0035] See Figure 2 , Figure 2 A flowchart of a data processing method provided in one embodiment of this specification is shown, which specifically includes the following steps.

[0036] Step 202: Input the target image containing at least one object to be detected into the detection model to obtain the detection bounding box for each object to be detected.

[0037] The object to be detected can be understood as a specific entity or region in the target image that needs to be classified, identified, or judged. In the context of homework grading, the object to be detected can be understood as various answer areas, such as multiple-choice question filling boxes, blank lines for fill-in-the-blank questions, and formula writing areas for calculation questions. The detection model can be understood as a computer vision model that has been trained on a large amount of data and can output the target position end-to-end. For example, the detection model is a deep learning-based model such as YOLO (You Only LookOnce) series or DETR (DEtection Transformer, object detection).

[0038] Specifically, the entire target image containing a complex background and the object to be detected is input into a pre-trained detection model. The detection model automatically outputs the precise coordinate bounding box (i.e., detection box) of each object to be detected in the target image. Of course, the detection model can also output the detection type corresponding to each detection box. For example, if the horizontal direction is detected within the detection box, the detection type corresponding to the detection box is determined to be a fill-in-the-blank question.

[0039] In practice, the target image is input into the detection model. The detection model extracts features from the target image to generate a multi-scale feature map. By making predictions on the feature map, a large number of candidate boxes are generated. A post-processing algorithm filters out overlapping and low-confidence predictions, outputting the final, high-confidence detection boxes. Each detection box is typically represented by normalized coordinates (such as the coordinates of the top left and bottom right corners, or the coordinates of the center point combined with width and height).

[0040] By replacing traditional manual selection or fixed template matching with detection models, data processing efficiency and generalization ability can be greatly improved. The models can intelligently adapt to images with different question layouts, shooting angles and lighting conditions, providing an accurate detection basis for subsequent precise grading.

[0041] Step 204: Construct prompt text based on the detection bounding box of each object to be detected, the reference result of each object to be detected, and at least one candidate determination type. The reference result is used to determine the type of the object to be detected, and the candidate determination type is represented by the basic semantic unit of the multimodal data processing model.

[0042] Among them, the reference result can be understood as the standard answer or basis for judging the object to be tested. In the scenario of homework correction, the reference answer is the reference answer of the test question. The candidate judgment type can be understood as the possible judgment conclusions corresponding to the object to be tested. For example, in the scenario of homework correction, the candidate judgment types include correct, incorrect, blank, partially correct, etc. The basic semantic unit of the multimodal large model can be understood as a token. The token is the smallest processing unit for the multimodal large model to process different modal data (such as text, image, sound, etc.). The specific model splits the input data into a token sequence, performs unified encoding and calculation. The token, as a unified data representation form in the model, can balance the differences between different modalities.

[0043] The detection bounding boxes (i.e. decision objects) of each object to be detected, the reference results (i.e. decision criteria) of each object to be detected, and at least one candidate decision type (i.e. decision target) are integrated into a structured prompt text rich in contextual semantics. This structured prompt text is used to accurately guide the multimodal large model to perform subsequent decision tasks.

[0044] In one or more embodiments of this specification, when constructing the prompt text, the position information of each object to be detected is determined using the detection bounding box of each object to be detected, and a structured target reference structure sequence is constructed based on the reference structure of each object to be detected. Thus, the prompt text is constructed based on the position information corresponding to each object to be detected, the target reference result sequence, at least one candidate judgment type, and processing rules. Specific implementation methods are described below: Based on the detection bounding box of each object to be detected, the reference result of each object to be detected, and at least one candidate judgment type, a prompt text is constructed, including: Determine the position information of the detection box corresponding to each object to be detected, and construct a target reference result sequence based on the reference results of each object to be detected; The prompt text is constructed based on the position information of the detection box corresponding to each object to be detected, the target reference result sequence, at least one candidate judgment type, and the processing rules, wherein the processing rules include the judgment logic for determining the judgment type corresponding to the object to be detected.

[0045] Specifically, the information is organized into a serialized sequence, which involves extracting the normalized coordinate information (i.e., position information, such as the coordinates of the top left and bottom right corners of the detection box) from the detection model, and organizing the reference results of multiple objects to be detected into a structured sequence of target reference results, such as separating the reference results of different objects to be detected with newline characters.

[0046] The position information of the detection boxes corresponding to the above-mentioned objects to be detected, the target reference result sequence, at least one preset candidate judgment type, a series of processing rules, and output specifications are all assembled into the final prompt text. The processing rules define the specific logic and constraints in the judgment process. For example, the processing rules stipulate that multi-line fill-in boxes should be regarded as a whole answer unit for evaluation, or require that the correction results output by the model must strictly correspond to the order of the input boxes. The output specifications require that the output structure of the correction results be complete, the format be uniform, and the semantics be clear. For example, the output specifications stipulate that the correction result sequence is separated by spaces. When the candidate judgment type is designed as the basic semantic unit in the multimodal large model vocabulary, each candidate judgment type corresponds to an independent identifier, so that the multimodal large model can directly output these predefined tokens with extremely high efficiency.

[0047] Taking the grading of a Chinese homework assignment as an example, the detection model obtains detection boxes corresponding to multiple fill-in-the-blank answer areas, acquires the position information of each detection box, and organizes the reference answers corresponding to each fill-in-the-blank answer area into a target reference result sequence, such as "1. less\nice, cold\nwater\n2. clothe\n3. example: divine blessing", where "\n" represents a newline character; and selects candidate judgment types (such as... <pass> , <failed> , <undone> ， <half>The system integrates processing rules (such as "multi-line fill-in boxes are treated as a single answer unit") and output specifications to obtain the final generated prompt text.

[0048] For example, the prompt text is: As an intelligent grading assistant, you need to accurately grade students' answers based on the reference information. Please strictly follow the guidelines below: Candidate Decision Type: 1. <pass>-The answer is completely correct. 2. <failed>-The answer is completely wrong. 3. <undone>- Blank / Scribbles 4. <half>- Partially correct 5. <uncertain>- Insufficient information to determine correctness 6. <ignore>- Meaningless and should be ignored Processing rules: 1. Multi-line fill-in-the-blank boxes are considered as a single answer unit. 2. Select the area where you need to edit the entire answer. 3. Repeated selection will only retain the most matching corrections. 4. Strictly maintain the correspondence between the correction boxes and the results. Output specifications: 1. Output only the space-separated sequence of correction results. 2. The number of results must strictly match the number of input boxes. Reference results: 1. less ice, cold water 2. clothes 3. example: mend Detection frame: <ai-check-boxroi="0.213,0.0450.248,0.085" / > <ai-check-boxroi="0.363,0.1370.410,0.175" / > <ai-check-boxroi="0.292,0.0890.370,0.130" / > <ai-check-boxroi="0.386,0.2650.534,0.316" / > <ai-check-boxroi="0.748,0.0360.888,0.089" / > <ai-check-boxroi="0.828,0.1200.967,0.178" / > " The data processing method provided in the embodiments of this specification transforms complex, batch-correction tasks into standardized processes that the model can execute precisely, significantly reducing ambiguity and arbitrary interpretation of large models, and ensuring the standardization of the judgment process and the structured nature of the output results.

[0049] Step 206: Input the target image and the prompt text into a multimodal data processing model to obtain a type sequence containing at least one target determination type, wherein the target determination type is determined from the at least one candidate determination type.

[0050] Among them, the multimodal data processing model, also known as the multimodal large model, is a large-scale pre-trained model that can simultaneously process and understand multimodal information such as images and text. Through learning from massive amounts of cross-modal data, it possesses powerful visual question answering, image description, and logical reasoning capabilities. The type sequence can be understood as an ordered set of judgment conclusions obtained by the multimodal data processing model after analyzing each object to be detected in the target image one by one.

[0051] Specifically, the target image containing the object to be detected and the prompt text constructed above are input into the multimodal data processing model. The multimodal data processing model is used to determine the type of each object to be detected, and finally outputs a type sequence consisting of at least one target determination type.

[0052] In one or more embodiments of this specification, a multimodal data processing model is used to obtain the object content of each object to be detected. The processing rules in the prompt text and the target reference result sequence are used to determine each object to be detected, resulting in a type sequence containing at least one target determination type. Specific implementation methods are described below: The target image and the prompt text are input into a multimodal data processing model to obtain a type sequence containing at least one target determination type, including: The target image and the prompt text are input into the multimodal data processing model. In the multimodal data processing model, the detection boxes of each object to be detected are used to locate each object and determine the object content of each object to be detected. Using the processing rules and the target reference result sequence, the object content of each object to be detected is judged to obtain a type sequence containing at least one target judgment type.

[0053] Specifically, a multimodal data processing model is used to perform multimodal information alignment and content understanding on the input target image and prompt text. When there is corresponding location information of the detection box in the prompt text, the location information of the detection box provided in the prompt text is used as spatial attention guidance, which can more accurately focus on each object to be detected in the target image, and then identify and interpret the specific content in the corresponding area of ​​the object to be detected, i.e., the object content, such as handwritten text, check mark or drawn graphic, etc., without limitation.

[0054] Based on the judgment logic in the processing rules, the multimodal data processing model compares and logically infers the content of the identified object with the target reference result sequence explicitly given in the prompt text to determine the most matching target judgment type for each object to be detected. When the target judgment type is determined from at least one candidate judgment type, the target judgment type is also the basic semantic unit of the multimodal data processing model. Therefore, the multimodal data processing model will output an ordered type sequence composed of basic semantic units according to the output specification.

[0055] Taking the grading of an English multiple-choice question as an example, the target image is the student's answer sheet, and the prompt text specifies the coordinates of five option boxes and the reference answer (i.e., the reference result) as "A, B, C, D, A". Upon receiving the target image and prompt text, the multimodal data processing model first locates the five option regions and identifies the student's actual selection as "A, B, D, D, A". The multimodal data processing model then compares each identification result (i.e., the object content) with the reference answer and applies processing rules (such as "a selection that exactly matches the answer is judged as correct") to make judgments sequentially, obtaining the type sequence output by the multimodal data processing model as "". <pass> <pass> <failed> <pass> <pass>".

[0056] The data processing method provided in the embodiments of this specification, when using a detection model to determine the corresponding detection box for each object to be detected and placing the position information of the detection box into the prompt text, the multimodal data processing model can quickly locate the object to be detected based on the position information of the detection box in the prompt text, thereby improving data processing efficiency and achieving more accurate recognition. Furthermore, the multimodal data processing model can achieve efficient end-to-end reasoning. Moreover, when outputting the target judgment type represented by basic semantic units, it greatly simplifies the complexity of result parsing, improves the processing speed and stability of the entire system, and enables real-time interactive applications.

[0057] Step 208: Match at least one target determination type in the type sequence with the at least one object to be detected to obtain the target determination type corresponding to each object to be detected.

[0058] Specifically, given the type sequence output by the multimodal data processing model, the target determination type in the type sequence is matched with the object to be detected to ensure that the target determination type can be accurately mapped to its corresponding object to be detected.

[0059] In practice, a set of ordered type sequences output by the multimodal data processing model are matched one-to-one with each detected object in the target image. Through a strict correspondence, each detected object marked by the detection box is assigned its corresponding target judgment type (i.e. judgment conclusion).

[0060] In one or more embodiments of this specification, when the number of target determination types in the type sequence is the same as the number of objects to be detected, a one-to-one matching can be performed directly according to the order of the detection boxes corresponding to the objects to be detected and the order of the target determination types in the type sequence to obtain the target determination type corresponding to each object to be detected. Specific implementation methods are as follows: Matching at least one target determination type in the type sequence with the at least one object to be detected to obtain the target determination type corresponding to each object to be detected, including: When the number of at least one target determination type in the type sequence is the same as the number of objects of the at least one object to be detected, the at least one target determination type in the type sequence is matched with the at least one object to be detected according to the order of the detection boxes corresponding to the at least one object to be detected and the order of at least one target determination type in the type sequence to obtain the target determination type corresponding to each object to be detected.

[0061] Specifically, the above embodiments mention that the prompt text contains an output specification, which stipulates that the number of results must be strictly consistent with the number of detection boxes, while the processing rules stipulate that the order of detection boxes and results must be maintained. Therefore, in an ideal situation, the number of at least one target determination type in the type sequence is consistent with the number of at least one object to be detected, and the order is corresponding. Therefore, according to the order, the first target determination type in the type sequence is assigned to the first object to be detected in the detection box list, the second target determination type in the type sequence is assigned to the second object to be detected in the detection box list, and so on, until each object to be detected is assigned a target determination type.

[0062] When there are multiple objects to be detected, the detection boxes of the multiple objects to be detected in the prompt text of the above embodiment are arranged in order. Therefore, when processing is carried out in strict accordance with the processing rules corresponding to the order, the order of multiple target determination types in the type sequence is consistent with the order of the detection boxes. Therefore, the target determination type corresponding to each object to be detected can be determined from the type sequence according to the order mapping.

[0063] For example, the detection model identifies four bounding boxes (i.e., answer boxes, numbered Box1 to Box4). The order of the bounding box positions in the prompt text follows the order of Box1 to Box4. Based on this order, the multimodal data processing model outputs a type sequence. <pass> <failed> <pass> <undone>During the result matching phase, the target judgment type in the type sequence is bound one by one according to the strict correspondence between Box1 and Box4, that is, Box1 is marked as... <pass>Box2 is marked as <failed>Box3 is marked as <pass>Box4 is marked as <undone>.

[0064] The data processing method provided in the embodiments of this specification, based on this sequential consistency strategy, greatly simplifies the complexity of system design and improves data processing efficiency, providing a solid data foundation for the visualization of subsequent results.

[0065] In one or more embodiments of this specification, when the number of at least one target determination type in the type sequence is inconsistent with the number of at least one object to be detected, i.e., when the model output is abnormal, the type sequence is processed to make the number of at least one target determination type in the obtained updated type sequence consistent with the number of at least one object to be detected. Specific implementation methods are as follows: Matching at least one target determination type in the type sequence with the at least one object to be detected to obtain the target determination type corresponding to each object to be detected, including: If the number of at least one target determination type in the type sequence is inconsistent with the number of objects of the at least one object to be detected, the type sequence is post-processed to obtain an updated type sequence, wherein the number of at least one target determination type in the updated type sequence is consistent with the number of objects of the at least one object to be detected; At least one target determination type in the update type sequence is matched with at least one object to be detected to obtain the target determination type corresponding to each object to be detected.

[0066] Post-processing can be understood as a set of preset data repair and correction rules, which are used to forcibly normalize data that does not meet expectations into data that meets the expected requirements.

[0067] Specifically, when there is a difference between the number of judgment conclusions output by the multimodal data processing model (i.e., the number of types of at least one target judgment type) and the number of objects to be detected identified by the detection model (i.e., the number of objects of at least one object to be detected), the number of judgment conclusions and the number of objects to be detected are realigned through a strategy of addition or deletion, thereby ensuring that each object to be detected can obtain a definite judgment conclusion.

[0068] In specific implementation, this post-processing logic mainly targets two scenarios. When the number of judgment conclusions output by the multimodal data processing model exceeds the number of objects to be detected, the post-processing of the type sequence is a reduction process. This process removes redundant target judgment types from the type sequence, obtaining an updated type sequence where the number of judgment conclusions matches the number of objects to be detected. When the number of judgment conclusions output by the multimodal data processing model is less than the number of objects to be detected, the post-processing of the type sequence is a supplementary process. This process adds preset target judgment types to the type sequence until the number of judgment conclusions in the updated type sequence matches the number of objects to be detected. After obtaining an updated type sequence with a consistent number of judgment conclusions, the above-mentioned sequential mapping principle is continued to assign the target judgment types in the updated type sequence to the corresponding objects to be detected, position by position. The specific implementation method is as follows: When the number of types of at least one target determination type in the type sequence is inconsistent with the number of objects of the at least one object to be detected, the type sequence is post-processed to obtain an updated type sequence, including: If the number of at least one target determination type in the type sequence is greater than the number of objects of the at least one object to be detected, redundant determination types are removed from the at least one target determination type in the type sequence to obtain the updated type sequence, wherein the redundant determination type is the determination type that is last in the type sequence. If the number of at least one target determination type in the type sequence is less than the number of objects of the at least one object to be detected, a supplementary determination type is added to the type sequence to obtain the updated type sequence.

[0069] Redundant decision types can be understood as one or more target decision types exceeding the required number located at the end of the sequence when the type sequence length exceeds the expectation; supplementary decision types can be understood as preset decision types used to fill the type sequence to the expected length when the output type length is insufficient, typically using type codes representing unknown, unprocessed, or neutral (e.g., ...). <uncertain>).

[0070] Specifically, when the length of the type sequence exceeds expectations or is insufficient, the type sequence output by the multimodal data processing model is forced to be strictly consistent with the number of objects to be detected determined by the detection model by removing from or adding to the end of the type sequence, thus laying the foundation for subsequent sequence matching.

[0071] In practice, when the number of types in the type sequence exceeds the number of objects to be detected, a truncation operation is performed. In real-world applications, multimodal data processing models may generate redundant outputs due to generation uncertainties. By default, these redundant outputs are attached to the end of the type sequence. Therefore, target judgment types are removed sequentially from the end of the type sequence. During the removal process, it is necessary to compare the number of types in the type sequence after removal with the number of objects to be detected. If the number of types matches the number of objects, the removal process is stopped. The type sequence obtained at this time is the updated type sequence.

[0072] When the number of types in the type sequence is less than the number of objects to be detected, a padding operation is performed. During the padding operation, a preset supplementary judgment type is selected for padding. When the detection boxes in the prompt text are arranged in order, the target judgment type in the type sequence corresponds to the detection box. Therefore, when the number of types in the type sequence is less than the number of objects to be detected, the position of the missing type in the type sequence can be determined, and the preset supplementary judgment type is added to the position of the missing type in the type sequence, thereby forming a usable updated type sequence.

[0073] In practical applications, the detection model identifies the five multiple-choice answer boxes on a student's answer sheet as the object to be detected, but the multimodal data processing model outputs a type sequence containing four target determination types. <pass> <failed> <pass> <pass>At this point, the number of types in the type sequence (4) is less than the number of objects to be detected (5), and it is determined that there is a missing value at the fourth position in the type sequence. Therefore, a supplementary judgment type (such as...) can be added at the fourth position of the type sequence. <uncertain>), generate update type sequence as <pass> <failed> <pass> <uncertain> <pass>This ensures that all five option boxes can obtain the corresponding judgment result.

[0074] The multimodal data processing model outputs a type sequence containing six target determination types. <pass> <failed> <pass> <pass> <failed> <pass>At this point, the number of types (6) in the type sequence is greater than the number of objects to be detected (5). Therefore, the type sequence can be truncated at the end to remove the last target judgment type and generate the updated type sequence. <pass> <failed> <pass> <pass> <failed>.

[0075] The data processing method provided in the embodiments of this specification ensures that a structurally complete sequence can still be provided when facing fluctuations in the output of multimodal data processing models through post-processing of the type sequence. This enhances the stability and fault tolerance of the system when facing unpredictable model behavior, prevents the failure of subsequent result display due to fluctuations in the output of multimodal data processing models, and provides reliable and regular data input for subsequent result display.

[0076] In one or more embodiments of this specification, after obtaining the target determination type corresponding to each detection object, correction information is generated based on the location information of the object to be detected and the corresponding target determination type. The correction information is then returned to the client that sent the correction request, so that the processed image generated based on the target image and the correction information can be displayed on the client. The specific implementation is as follows: Before inputting a target image containing at least one object to be detected into the detection model to obtain the detection bounding box for each object, the process includes: Receive a modification request sent by the client, wherein the modification request carries the target image; After matching at least one target determination type in the type sequence with the at least one object to be detected to obtain the target determination type corresponding to each object to be detected, the method further includes: Based on the target determination type corresponding to each object to be detected and the position information of the detection box corresponding to each object to be detected, batch correction information is generated. The correction information is returned to the client so that the processed image generated based on the target image and the correction information can be displayed on the client's user interface.

[0077] The client can be understood as any terminal application, such as a web page front-end, mobile application, or desktop software; the correction information can be understood as a structured data carrier that contains the judgment conclusion (i.e. target judgment type) of each object to be detected and its position coordinates in the target image.

[0078] Specifically, upon receiving a correction request from the client, the system parses the target image to be corrected carried in the correction request, performs complex detection and reasoning through the detection model and multimodal data processing model, and obtains the target judgment type corresponding to each object to be detected after matching.

[0079] Instead of directly returning a type sequence containing the target determination type of each object to be detected, the system generates a structured correction information based on the target determination type of each object and the position information of its detection box. This correction information clearly records "which position (coordinates) of the object was determined to what result (type)". This correction information is returned to the client that initiated the correction request. Based on the correction information and the target image, a processed image carrying correction traces can be generated. The objects to be detected in this processed image are surrounded by detection boxes, and the detection boxes are different colors. Each color indicates a target determination type. For example, if the target determination type is correct, the detection box is represented by green, and if the target determination type is incorrect, the detection box is represented by red.

[0080] In practical applications, teachers can upload a student's homework image via a web client (front-end) and click the "Grade" button, triggering a grading request sent to the server containing the student's homework image (i.e., the target image). After the server completes all processing, it generates grading information, for example, the grading information includes: {box_id:1,roi:[0.1,0.2,0.15,0.25],judgment: <pass>},{box_id:2,roi:[0.3,0.4,0.35,0.45],judgment: <failed>}, where box_id is the detection box identifier, roi is used to represent the position coordinates of the detection box, and judgment is the target determination type of the object to be detected.

[0081] After receiving this correction information, the client uses the position coordinates and target determination type in the correction information to perform overlay rendering on the target image. For example, it displays a green detection box on the first box and a red detection box on the second box, thereby generating and displaying the final processed image.

[0082] The data processing method provided in the embodiments of this specification is such that the client is responsible for user experience and interactive display, while the server is used to provide stable and accurate data services. This interactive architecture greatly improves the scalability and availability of the system and supports diverse client integration, thereby improving the user experience.

[0083] The data processing method provided in the embodiments of this specification, through the design of pre-detection bounding boxes and corresponding prompt text for grading tasks, achieves both fine-grained grading and positioning of each bounding box and efficient real-time inference. Based on the powerful image and semantic understanding capabilities of the multimodal data processing model, it realizes a highly versatile data processing method applicable to homework grading scenarios, supporting various question types (fill-in-the-blank, short answer, graphing, etc.). In other words, by utilizing the image and semantic understanding capabilities of the multimodal data processing model, combined with efficient bounding box detection and task format design, it achieves a highly versatile, real-time, and accurate data processing method.

[0084] See Figure 3 , Figure 3 A flowchart illustrating the processing procedure of a data processing method according to an embodiment of this specification is shown, specifically including the following steps.

[0085] Step 302: Input images of students' answers and reference results.

[0086] The system receives a grading request from the client. The grading request includes an image of the student's answer to the question to be graded (i.e., the target image in the above embodiment). The grading request may also carry reference results (i.e., reference answers) provided in the form of text, images, or a combination of text and images. If no reference answer is provided, the system enters the no-reference-standard mode.

[0087] Step 304: Detect the answer box.

[0088] The input images of students answering questions are processed using pre-trained detection models (such as YOLO model and DETR model).

[0089] The detection model automatically locates the answer region to be analyzed in the image (i.e., the object to be detected mentioned above) and outputs a precise answer box (i.e., the detection box mentioned above) for each answer region. Each detection box not only contains its normalized position coordinates in the image, but is also assigned an answer type label, such as multiple choice, true or false, fill-in-the-blank, arithmetic, grid, check mark, connect line, drawing, or word problem, etc.

[0090] Of course, in this embodiment, a multimodal large model can also be used to directly predict the answer box, but this method involves a large number of tokens and high inference time, which is not suitable for real-time grading. Therefore, the YOLO model with better real-time performance is selected as the detection model.

[0091] Step 306: Generate prompt text.

[0092] The detected answer box location information, reference results, at least one candidate judgment type, specific processing rules for different question types, and output rules are integrated into a structured prompt text.

[0093] In fact, the prompt text can contain the question type corresponding to the answer box, so that the subsequent multimodal large model can make accurate judgments based on the accurate question type and the specific processing rules of the corresponding question type.

[0094] It should be noted that the candidate decision type is designed as a single token, which is added to the model vocabulary, for example: <pass> 、 <failed> 、 <undone> 、 <half> 、 <uncertain> 、 <ignore>That is, each token represents a judgment type (e.g., correct, incorrect, no answer, half correct, uncertain, ignored).

[0095] Step 308: Perform inference based on the multimodal large model.

[0096] The multimodal large model (i.e., the multimodal data processing model mentioned above) is a model processed by SFT. The target image and the generated structured prompt text are input into the multimodal large model. Leveraging its powerful cross-modal understanding capabilities, the multimodal large model performs content recognition and logical interpretation on each specified response region based on the instructions, reference results, and location coordinates in the prompt text. This multimodal large model does not output freely generated text; instead, it directly generates a type sequence composed of the aforementioned custom tokens. For example, the type sequence output by the model might be... <failed> <pass> <pass> <pass> <pass> <pass>This serialized output method can greatly improve reasoning efficiency.

[0097] Step 310: Result analysis and post-processing.

[0098] The type sequence output by the multimodal large model is parsed and matched sequentially with the response boxes detected in step 304. Response boxes with the type "ignore" are ignored.

[0099] Specifically, the length of the output type sequence is strictly checked to ensure it matches the number of answer boxes. If the type sequence is too long, redundant results are automatically truncated from the end; if the sequence is too short, preset default categories (such as...) are added to the type sequence. <uncertain>) to fill in the number of answers, to ensure that the length of the output type sequence is consistent with the number of answer boxes in the input.

[0100] Step 312: Client marking display.

[0101] According to the position coordinates of each answer box and the corresponding correction result (i.e. target determination type), a structured correction information is generated, and the correction information is returned to the client that initiates the correction request.

[0102] When the client receives the correction information, the coordinate information is used to perform overlay rendering on the original student answer question image, and different colored detection boxes are marked to obtain a processed image. See Figure 4 , Figure 4 FIG. 1 shows a schematic diagram of a processed image in a data processing method according to an embodiment of the present specification.

[0103] Of course, a check mark, a cross or a half check symbol can be dynamically superimposed beside the answer box, so that the correction result can be clearly and intuitively displayed on the user interaction interface, which is not limited herein.

[0104] The data processing method provided by the embodiment of the present specification uses a detection model to quickly complete answer box detection, and combines a self-defined single token correction output to significantly improve the real-time performance of data processing. The SFT training and post-processing strategy based on the structured prompt text and the output format design ensures that the order of the multi-box correction result is consistent, and ensures that the output result is reliable and can be stably corresponding to the answer box. The position information, question type, reference answer and other multi-dimensional information are used to improve the accuracy of the output result, and the multi-modal large model can support multiple question types, including drawing questions and open questions that cannot be OCR, to realize a data processing method with strong universality.

[0105] Corresponding to the above method embodiments, the present specification also provides data processing device embodiments, Figure 5 FIG. 1 shows a structural schematic diagram of a data processing device according to an embodiment of the present specification. As shown in Figure 5 The device comprises: The detection module 502 is configured to input a target image containing at least one to-be-detected object into a detection model to obtain a detection box of each to-be-detected object; The construction module 504 is configured to construct a prompt text according to the detection box of each to-be-detected object, a reference result of each to-be-detected object, and at least one candidate determination type, wherein the reference result is used to determine the type of the to-be-detected object, and the candidate determination type is represented by a basic semantic unit of a multi-modal data processing model; The obtaining module 506 is configured to input the target image and the prompt text into a multi-modal data processing model to obtain a type sequence containing at least one target determination type, where the target determination type is determined from the at least one candidate determination type. The matching module 508 is configured to match the at least one target determination type in the type sequence with the at least one to-be-detected object to obtain a target determination type corresponding to each to-be-detected object.

[0106] Optionally, the constructing module 504 is further configured to: determine position information of a detection frame corresponding to each to-be-detected object, and construct a target reference result sequence according to a reference result of each to-be-detected object; construct the prompt text according to the position information of each to-be-detected object, the target reference result sequence, at least one candidate determination type, and a processing rule, where the processing rule contains determination logic for determining a determination type corresponding to the to-be-detected object.

[0107] Optionally, the obtaining module 506 is further configured to: input the target image and the prompt text into the multi-modal data processing model, position each to-be-detected object by using a detection frame of each to-be-detected object in the multi-modal data processing model, and determine an object content of each to-be-detected object; determine the object content of each to-be-detected object by using the processing rule and the target reference result sequence to obtain a type sequence containing at least one target determination type.

[0108] Optionally, the matching module 508 is further configured to: in a case where a type number of the at least one target determination type in the type sequence is consistent with an object number of the at least one to-be-detected object, match the at least one target determination type in the type sequence with the at least one to-be-detected object according to an order of a detection frame corresponding to the at least one to-be-detected object and an order of the at least one target determination type in the type sequence to obtain a target determination type corresponding to each to-be-detected object.

[0109] Optionally, the matching module 508 is further configured to: in a case where a type number of the at least one target determination type in the type sequence is inconsistent with an object number of the at least one to-be-detected object, perform post-processing on the type sequence to obtain an updated type sequence, where a type number of at least one target determination type in the updated type sequence is consistent with the object number of the at least one to-be-detected object. Match at least one target determination type in the sequence of the updated types with the at least one object to be detected, to obtain a target determination type corresponding to each object to be detected.

[0110] Optionally, the matching module 508 is further configured to: In a case where a type number of the at least one target determination type in the sequence of the types is greater than an object number of the at least one object to be detected, remove a redundant determination type from the at least one target determination type in the sequence of the types, to obtain the sequence of the updated types, wherein the redundant determination type is a determination type at a last position in the sequence of the types. In a case where a type number of the at least one target determination type in the sequence of the types is less than an object number of the at least one object to be detected, add a supplementary determination type to the sequence of the types, to obtain the sequence of the updated types.

[0111] The apparatus further includes: The receiving module is configured to receive a correction request sent by a client, wherein the correction request carries the target image.

[0112] The apparatus further includes: The response module is configured to generate correction information according to the target determination type corresponding to each object to be detected and position information of a detection frame corresponding to each object to be detected, and return the correction information to the client, so as to display, on a user interactive interface of the client, a processing image generated based on the target image and the correction information.

[0113] The above is a schematic scheme of the data processing apparatus of the embodiment. It should be noted that the technical scheme of the data processing apparatus and the technical scheme of the data processing method described above belong to the same concept, and the details of the technical scheme of the data processing apparatus which are not described in detail can be referred to the description of the technical scheme of the data processing method.

[0114] Figure 6 A structural block diagram of a computing device 600 is shown according to an embodiment of the present specification. The components of the computing device 600 include but are not limited to a memory 610 and a processor 620. The processor 620 is connected to the memory 610 through a bus 630, and a database 650 is used to save data.

[0115] The computing device 600 also includes an access device 640, which enables the computing device 600 to communicate via one or more networks 660. Examples of these networks include Public Switched Telephone Network (PSTN), Local Area Network (LAN), Wide Area Network (WAN), Personal Area Network (PAN), or combinations of communication networks such as the Internet. The access device 640 may include one or more of any type of wired or wireless network interface (e.g., a network interface card (NIC)), such as an IEEE 802.11 Wireless Local Area Network (WLAN) wireless interface, a Wi-MAX (Worldwide Interoperability for Microwave Access) interface, an Ethernet interface, a Universal Serial Bus (USB) interface, a cellular network interface, a Bluetooth interface, or a Near Field Communication (NFC) interface.

[0116] In one embodiment of this specification, the above-described components of the computing device 600 and Figure 6 Other components, not shown, can also be connected to each other, for example, via a bus. It should be understood that... Figure 6 The block diagram of the computing device shown is for illustrative purposes only and is not intended to limit the scope of this specification. Those skilled in the art can add or replace other components as needed.

[0117] The computing device 600 can be any type of stationary or mobile computing device, including mobile computers or mobile computing devices (e.g., tablet computers, personal digital assistants, laptop computers, notebook computers, netbooks, etc.), mobile phones (e.g., smartphones), wearable computing devices (e.g., smartwatches, smart glasses, etc.) or other types of mobile devices, or stationary computing devices such as desktop computers or personal computers (PCs). The computing device 600 can also be a mobile or stationary server.

[0118] The processor 620 is used to execute the following computer program / instructions, which, when executed by the processor, implement the steps of the above-described data processing method.

[0119] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the computing device embodiments are basically similar to the data processing method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions of the data processing method embodiments.

[0120] An embodiment of this specification also provides a computer-readable storage medium storing a computer program / instructions that, when executed by a processor, implement the steps of the above-described data processing method.

[0121] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the computer-readable storage medium embodiments are basically similar to the data processing method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions of the data processing method embodiments.

[0122] An embodiment of this specification also provides a computer program product, including a computer program / instructions that, when executed by a processor, implement the steps of the above-described data processing method.

[0123] The above is an illustrative scheme of a computer program product according to this embodiment. It should be noted that the technical solution of this computer program product and the technical solution of the data processing method described above belong to the same concept. For details not described in detail in the technical solution of the computer program product, please refer to the description of the technical solution of the data processing method described above.

[0124] The foregoing has described specific embodiments of this specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims may be performed in a different order than that shown in the embodiments and may still achieve the desired result. Furthermore, the processes depicted in the drawings do not necessarily require the specific or sequential order shown to achieve the desired result. In some embodiments, multitasking and parallel processing are possible or may be advantageous.

[0125] The computer instructions include computer program code, which may be in the form of source code, object code, executable file, or certain intermediate forms. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording media, USB flash drive, portable hard drive, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc. It should be noted that the content included in the computer-readable medium may be appropriately added or removed according to the requirements of patent practice. For example, in some regions, according to patent practice, computer-readable media may not include electrical carrier signals and telecommunication signals.

[0126] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that the embodiments in this specification are not limited to the described order of actions, because according to the embodiments in this specification, some steps can be performed in other orders or simultaneously. Furthermore, those skilled in the art should also understand that the embodiments described in this specification are all preferred embodiments, and the actions and modules involved are not necessarily essential to the embodiments in this specification.

[0127] In the above embodiments, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0128] The preferred embodiments disclosed above are merely illustrative of this specification. The optional embodiments do not exhaustively describe all details, nor do they limit the invention to the specific implementations described. Clearly, many modifications and variations can be made based on the embodiments described herein. These embodiments are selected and specifically described in this specification to better explain the principles and practical applications of the embodiments, thereby enabling those skilled in the art to better understand and utilize this specification. This specification is limited only by the claims and their full scope and equivalents.< / uncertain> < / pass> < / pass> < / pass> < / pass> < / pass> < / failed> < / ignore> < / uncertain> < / half> < / undone> < / failed> < / pass> < / failed> < / pass> < / failed> < / pass> < / pass> < / failed> < / pass> < / pass> < / failed> < / pass> < / pass> < / failed> < / pass> < / pass> < / uncertain> < / pass> < / failed> < / pass> < / uncertain> < / pass> < / pass> < / failed> < / pass> < / uncertain> < / undone> < / pass> < / failed> < / pass> < / undone> < / pass> < / failed> < / pass> < / pass> < / pass> < / failed> < / pass> < / pass> < / ignore> < / uncertain> < / half> < / undone> < / failed> < / pass> < / half> < / undone> < / failed> < / pass>

Claims

1. A data processing method, characterized in that, include: Input the target image containing at least one object to be detected into the detection model to obtain the detection bounding box for each object to be detected. Based on the detection bounding box of each object to be detected, the reference result of each object to be detected, and at least one candidate determination type, a prompt text is constructed, wherein the reference result is used to determine the type of the object to be detected, and the candidate determination type is represented by the basic semantic unit of the multimodal data processing model. The target image and the prompt text are input into a multimodal data processing model to obtain a type sequence containing at least one target determination type, wherein the target determination type is determined from the at least one candidate determination type; At least one target determination type in the type sequence is matched with at least one object to be detected to obtain the target determination type corresponding to each object to be detected.

2. The method as described in claim 1, characterized in that, Based on the detection bounding box of each object to be detected, the reference result of each object to be detected, and at least one candidate judgment type, a prompt text is constructed, including: Determine the position information of the detection box corresponding to each object to be detected, and construct a target reference result sequence based on the reference results of each object to be detected; The prompt text is constructed based on the position information of the detection box corresponding to each object to be detected, the target reference result sequence, at least one candidate judgment type, and the processing rules, wherein the processing rules include the judgment logic for determining the judgment type corresponding to the object to be detected.

3. The method as described in claim 2, characterized in that, The target image and the prompt text are input into a multimodal data processing model to obtain a type sequence containing at least one target determination type, including: The target image and the prompt text are input into the multimodal data processing model. In the multimodal data processing model, the detection boxes of each object to be detected are used to locate each object and determine the object content of each object to be detected. Using the processing rules and the target reference result sequence, the object content of each object to be detected is judged to obtain a type sequence containing at least one target judgment type.

4. The method as described in claim 1, characterized in that, Matching at least one target determination type in the type sequence with the at least one object to be detected to obtain the target determination type corresponding to each object to be detected, including: When the number of at least one target determination type in the type sequence is the same as the number of objects of the at least one object to be detected, the at least one target determination type in the type sequence is matched with the at least one object to be detected according to the order of the detection boxes corresponding to the at least one object to be detected and the order of at least one target determination type in the type sequence to obtain the target determination type corresponding to each object to be detected.

5. The method as described in claim 1, characterized in that, Matching at least one target determination type in the type sequence with the at least one object to be detected to obtain the target determination type corresponding to each object to be detected, including: If the number of at least one target determination type in the type sequence is inconsistent with the number of objects of the at least one object to be detected, the type sequence is post-processed to obtain an updated type sequence, wherein the number of at least one target determination type in the updated type sequence is consistent with the number of objects of the at least one object to be detected; At least one target determination type in the update type sequence is matched with at least one object to be detected to obtain the target determination type corresponding to each object to be detected.

6. The method as described in claim 5, characterized in that, When the number of types of at least one target determination type in the type sequence is inconsistent with the number of objects of the at least one object to be detected, the type sequence is post-processed to obtain an updated type sequence, including: If the number of at least one target determination type in the type sequence is greater than the number of objects of the at least one object to be detected, redundant determination types are removed from the at least one target determination type in the type sequence to obtain the updated type sequence, wherein the redundant determination type is the determination type that is last in the type sequence. If the number of at least one target determination type in the type sequence is less than the number of objects of the at least one object to be detected, a supplementary determination type is added to the type sequence to obtain the updated type sequence.

7. The method as described in claim 1, characterized in that, Before inputting a target image containing at least one object to be detected into the detection model to obtain the detection bounding box for each object, the process includes: Receive a modification request sent by the client, wherein the modification request carries the target image; After matching at least one target determination type in the type sequence with the at least one object to be detected to obtain the target determination type corresponding to each object to be detected, the method further includes: Based on the target determination type corresponding to each object to be detected and the position information of the detection box corresponding to each object to be detected, batch correction information is generated. The correction information is returned to the client so that the processed image generated based on the target image and the correction information can be displayed on the client's user interface.

8. A data processing apparatus, characterized in that, include: The detection module is configured to input a target image containing at least one object to be detected into the detection model to obtain detection bounding boxes for each object to be detected. The construction module is configured to construct prompt text based on the detection box of each object to be detected, the reference result of each object to be detected, and at least one candidate determination type, wherein the reference result is used to determine the type of the object to be detected, and the candidate determination type is represented by the basic semantic unit of the multimodal data processing model. The acquisition module is configured to input the target image and the prompt text into a multimodal data processing model to obtain a type sequence containing at least one target determination type, wherein the target determination type is determined from the at least one candidate determination type; The matching module is configured to match at least one target determination type in the type sequence with the at least one object to be detected, thereby obtaining the target determination type corresponding to each object to be detected.

9. A computing device, characterized in that, include: Memory and processor; The memory is used to store computer programs / instructions, and the processor is used to execute the computer programs / instructions, which, when executed by the processor, implement the steps of the method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, It stores a computer program / instructions that, when executed by a processor, implement the steps of the method according to any one of claims 1 to 7.

11. A computer program product, characterized in that, Includes a computer program / instructions that, when executed by a processor, implement the steps of the method according to any one of claims 1 to 7.

Images