Content labeling method and content labeling model training method

Abstract

The embodiment of the specification provides a content marking method and a content marking model training method, wherein the content marking method comprises the following steps: obtaining to-be-marked content and a target marking intention of the to-be-marked content; inputting the target marking intention and the to-be-marked content into a content marking model to obtain marking position information, wherein the content marking model is trained based on sample marked content, a sample marking intention of the sample marked content and sample position information matched with the sample marking intention, and the sample marking intention corresponds to at least two different sample visual marking forms; and marking the to-be-marked content based on the marking position information to obtain a content marking result. By introducing the target marking intention and fusing sample training of multi-form visual marking, adaptive marking position information is automatically generated, intelligent upgrading from single fixed style marking to intention driving and multi-state adaptation is realized, and the accuracy and flexibility of marking are significantly improved.

Description

Technical Field

[0001] The embodiments in this specification relate to the field of artificial intelligence technology, and in particular to content tagging methods and content tagging model training methods. Background Technology

[0002] With the deep integration of artificial intelligence and educational technology, intelligent homework grading systems have been widely used in basic education, significantly improving teaching efficiency and the timeliness of feedback. Marking homework, test papers, and other content within the intelligent homework grading process is gradually becoming a research focus.

[0003] Currently, the main approach is to label specific physical entities within the content to be labeled, such as circling the cat in an image. However, this method struggles to understand complex natural language instructions, and the labeling results fail to meet users' needs, thus hindering the widespread application of intelligent homework grading systems. Summary of the Invention

[0004] In view of this, embodiments of this specification provide a content tagging method. One or more embodiments of this specification also relate to a content tagging model training method, a content tagging apparatus, a content tagging model training apparatus, a computing device, a computer-readable storage medium, and a computer program product, to address the technical deficiencies existing in the prior art.

[0005] According to a first aspect of the embodiments of this specification, a content tagging method is provided, comprising: acquiring content to be tagged and a target tagging intent for the content to be tagged; inputting the target tagging intent and the content to be tagged into a content tagging model to obtain tagging location information, wherein the content tagging model is trained based on sample tagging content, sample tagging intent of the sample tagging content, and sample location information matching the sample tagging intent, and the sample tagging intent corresponds to at least two different sample visual tagging forms; and tagging the content to be tagged based on the tagging location information to obtain a content tagging result.

[0006] According to a second aspect of the embodiments of this specification, a content tagging model training method is provided, comprising: acquiring sample data, wherein the sample data includes sample tagging content, sample tagging intent of the sample tagging content, and sample location information matching the sample tagging intent, the sample tagging intent corresponding to at least two different sample visual tagging forms; inputting the sample tagging content and sample tagging intent into a content tagging model to obtain predicted location information; and adjusting the parameters of the content tagging model according to the sample location information and the predicted location information to obtain a trained content tagging model.

[0007] According to a third aspect of the embodiments of this specification, a content tagging device is provided, comprising: a first acquisition module configured to acquire content to be tagged and a target tagging intent for the content to be tagged; a first input module configured to input the target tagging intent and the content to be tagged into a content tagging model to obtain tagging location information, wherein the content tagging model is trained based on sample tagging content, sample tagging intent of the sample tagging content, and sample location information matching the sample tagging intent, and the sample tagging intent corresponds to at least two different sample visual tagging forms; and a tagging module configured to tag the content to be tagged based on the tagging location information to obtain a content tagging result.

[0008] According to a fourth aspect of the embodiments of this specification, a content tagging model training apparatus is provided, comprising: a second acquisition module configured to acquire sample data, wherein the sample data includes sample tagging content, sample tagging intent of the sample tagging content, and sample position information matching the sample tagging intent, the sample tagging intent corresponding to at least two different sample visual tagging forms; a second input module configured to input the sample tagging content and sample tagging intent into a content tagging model to obtain predicted position information; and an adjustment module configured to adjust the parameters of the content tagging model according to the sample position information and the predicted position information to obtain a trained content tagging model.

[0009] According to a fifth aspect of the embodiments of this specification, a computing device is provided, including: a memory and a processor; the memory is used to store computer programs / instructions, and the processor is used to execute the computer programs / instructions, wherein the computer programs / instructions, when executed by the processor, implement the steps of the methods provided in the first or second aspect described above.

[0010] According to a sixth aspect of the embodiments of this specification, a computer-readable storage medium is provided that stores a computer program / instructions that, when executed by a processor, implement the steps of the method provided in the first or second aspect described above.

[0011] According to a seventh aspect of the embodiments of this specification, a computer program product is provided, including a computer program / instructions that, when executed by a processor, implement the steps of the method provided in the first or second aspect described above.

[0012] This specification provides a content tagging method according to one embodiment, comprising: acquiring content to be tagged and a target tagging intent for the content to be tagged; inputting the target tagging intent and the content to be tagged into a content tagging model to obtain tagging position information, wherein the content tagging model is trained based on sample tagging content, sample tagging intent of the sample tagging content, and sample position information matching the sample tagging intent, and the sample tagging intent corresponds to at least two different sample visual tagging forms; and tagging the content to be tagged based on the tagging position information to obtain content tagging results. By introducing a target tagging intent and integrating sample training of multi-form visual tags, the content tagging model can accurately understand user tagging requirements and automatically generate appropriate tagging position information; it achieves an intelligent upgrade from "single fixed style tagging" to "intent-driven, multi-form adaptation", significantly improving the accuracy and flexibility of tagging, while greatly reducing the operational cost of manually locating and adjusting tagging forms one by one, providing an efficient and unified automated solution for batch and multi-type content tagging tasks. Attached Figure Description

[0013] Figure 1 This is a flowchart illustrating a content tagging method provided in one embodiment of this specification; Figure 2 This is a flowchart illustrating the processing steps of an image region drawing method provided in one embodiment of this specification. Figure 3 This is a flowchart illustrating a content tagging model training method provided in one embodiment of this specification; Figure 4 This is an architecture diagram of a content tagging system provided in one embodiment of this specification; Figure 5 This is a schematic diagram of the structure of a content marking device provided in one embodiment of this specification; Figure 6 This is a schematic diagram of the structure of a content tagging model training device provided in one embodiment of this specification; Figure 7 This is a structural block diagram of a computing device provided in one embodiment of this specification. Detailed Implementation

[0014] Many specific details are set forth in the following description to provide a full understanding of this specification. However, this specification can be implemented in many other ways than those described herein, and those skilled in the art can make similar extensions without departing from the spirit of this specification. Therefore, this specification is not limited to the specific implementations disclosed below.

[0015] The terminology used in one or more embodiments of this specification is for the purpose of describing particular embodiments only and is not intended to limit the scope of one or more embodiments of this specification. The singular forms “a,” “described,” and “the” as used in one or more embodiments of this specification and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used in one or more embodiments of this specification refers to and includes any or all possible combinations of one or more associated listed items. The term “at least one” as used in one or more embodiments of this specification means “one or more,” and “a plurality of” means “two or more.” The term “comprising” is an open-ended description and should be understood as “including but not limiting,” and may include other content in addition to what has been described.

[0016] It should be understood that although the terms first, second, etc., may be used to describe various information in one or more embodiments of this specification, such information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, first may also be referred to as second without departing from the scope of one or more embodiments of this specification, and similarly, second may also be referred to as first. Depending on the context, the word "if" as used herein may be interpreted as "when," "when," or "in response to a determination."

[0017] Furthermore, it should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in one or more embodiments of this specification are all information and data authorized by the user or fully authorized by all parties. Moreover, the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and corresponding operation entry points are provided for users to choose to authorize or refuse.

[0018] In one or more embodiments of this specification, a large model refers to a deep learning model with a large number of model parameters, typically containing hundreds of millions, tens of billions, hundreds of billions, trillions, or even tens of trillions of model parameters. A large model can also be called a foundation model. It is pre-trained using large-scale unlabeled corpora to produce a pre-trained model with hundreds of millions of parameters. Such models can adapt to a wide range of downstream tasks and have good generalization ability. Examples include Large Language Models (LLMs) and multi-modal pre-training models.

[0019] In practical applications, large models only require a small number of samples to fine-tune the pre-trained model before they can be applied to different tasks. Large models can be widely used in fields such as Natural Language Processing (NLP) and Computer Vision. Specifically, they can be applied to computer vision tasks such as Visual Question Answering (VQA), Image Captioning (IC), and Image Generation, as well as natural language processing tasks such as text-based sentiment classification, text summarization, and machine translation. The main application scenarios of large models include digital assistants, intelligent robots, search, online education, office software, e-commerce, and intelligent design.

[0020] First, the terms and concepts used in one or more embodiments of this specification will be explained.

[0021] Multimodal Large Language Model (MLLM): refers to a deep learning model that can simultaneously process image and text inputs and perform cross-modal understanding and generation.

[0022] Visual grounding: A technique for locating specific objects or regions in an image based on natural language descriptions, typically outputting a bounding box.

[0023] Circling: In the embodiments of this specification, it specifically refers to the operation of drawing visual marks on an image according to the marking intent.

[0024] Supervised Fine-Tuning (SFT): Training a pre-trained model using labeled "image-instruction-coordinate" data.

[0025] Bounding Box (BBox): Usually represented by four values ​​[x1, y1, x2, y2], which represent the coordinates of the top left and bottom right corners of the rectangle, respectively.

[0026] Token: In the fields of natural language processing and machine learning, it is the smallest semantic or processing unit. It refers to the independent fragment after the original text is split by a word segmentation algorithm. It can be a word, subword, character or punctuation mark.

[0027] Faster R-CNN is an end-to-end object detection model that introduces a region proposal network to integrate candidate region generation, feature extraction, object classification, and bounding box regression into a single deep neural network, achieving high-precision real-time object detection.

[0028] YOLO: It reconstructs object detection as an end-to-end regression problem. It divides the input image into a grid, and each grid simultaneously predicts the bounding box coordinates, confidence score, and class probability. It can directly map the detection results of all objects from the image pixels with only one forward propagation.

[0029] Optical Character Recognition (OCR) is a technology that converts printed or handwritten text in an image into machine-editable text. Its workflow typically includes image preprocessing (grayscale conversion, binarization, noise reduction), text region detection, character segmentation, feature extraction and classification, and finally outputs a text string.

[0030] Natural Language Processing (NLP) is an interdisciplinary field of artificial intelligence and linguistics that aims to enable computers to understand, parse, and generate human language, and to achieve natural interaction between humans and machines.

[0031] OpenCV: A cross-platform computer vision and machine learning software library that provides over 2,500 optimized algorithm interfaces, covering core functions such as image preprocessing (filtering, geometric transformation), feature detection (edges, corners), object tracking, face recognition, camera calibration, and deep learning model deployment.

[0032] JSON: A lightweight, programming language-independent data exchange format based on JavaScript syntax but supported by the vast majority of programming languages; its core structure consists of key-value pairs and ordered lists.

[0033] To achieve precise positioning within the content to be labeled, the industry-standard technical approach is visual positioning. The common practice is to discretize the coordinate values ​​(i.e., the label location information) into special tokens, allowing the model to generate a coordinate sequence similar to generating text.

[0034] Traditional object detection models often use Faster R-CNN or YOLO, which can detect predefined categories (such as "people" and "vehicles"). Traditional object detection schemes have limited category detection capabilities and lack intent understanding. They can only detect predefined, fixed categories (such as "people" and "boxes") and cannot handle open-set targets. When faced with complex intents described in natural language (such as "circle the second word in the third line"), due to the lack of a text encoder and multimodal alignment capabilities, these schemes cannot establish a connection between the intent and the content to be labeled, thus failing to complete the localization task.

[0035] OCR-based matching schemes are a common method for processing plain text content to be labeled. First, an OCR model is used to detect and recognize all text and its coordinates within the content to be labeled. Then, NLP techniques or rule-based matching algorithms are used to analyze keywords in the user's intent. Finally, the matching text coordinates are retrieved from the OCR-extracted results and a bounding box is selected. OCR technology is essentially text recognition; for non-text targets such as blank answer areas or geometric figures, this scheme fails because it cannot extract text features. Furthermore, this scheme relies heavily on keyword matching and struggles to understand generalized intents that require combining visual information with deep semantic reasoning, such as "circle the third character in the second line" or "circle the incorrect condition," resulting in a low localization success rate.

[0036] With the development of multimodal technology, general-purpose multimodal large language models have acquired strong image and text understanding capabilities. Visual localization schemes based on these models utilize MLLMs trained on large-scale image-text alignment for end-to-end inference. This approach directly inputs the content to be labeled and the target labeling intent. The model, through the collaborative work of a visual encoder and a language decoder, outputs the bounding box coordinates of the target object (i.e., the labeling location information). These general-purpose models are typically trained on public datasets (such as COCO and RefCOCO), tend to recognize specific visual entities, and their default output is often a single rectangular box. Existing general-purpose multimodal models are mainly trained on natural scene data (such as COCO), lacking an understanding of document structure and teaching logic. When faced with labeling intents containing complex business logic, such as "draw a box to the right of '2m' in the answer area," the model struggles to accurately distinguish between the question stem, options, and the answer area, easily leading to localization bias or illusions.

[0037] Current technologies primarily target specific physical entities (such as "circling the cat in the picture"). However, in educational and document processing scenarios, users often need to locate "blank answer areas" or "text fragments with specific meanings," and existing models lack the ability to locate these abstract targets. Related visual localization models typically only output bounding boxes. However, in educational settings, text marking often requires "underlines" rather than "bounding boxes" to conform to reading habits. Current technologies cannot distinguish between these two forms based on the marking intent.

[0038] The embodiments in this specification mainly address the technical problem that related visual positioning technologies cannot accurately and diversely visually mark (select or draw lines) various forms of targets (such as text lines and blank areas) based on complex natural language instructions (i.e., target marking intentions, especially abstract intentions in teaching scenarios).

[0039] This specification's embodiments aim to address the problem that general multimodal models output a single form and cannot respond to differentiated interactive intents ("box selection" and "line drawing"). By constructing data of different morphological instructions (i.e., sample marking intents), and establishing a correlation between instruction semantics and coordinate geometric attributes (i.e., sample visual marking form, such as aspect ratio and relative position), the model has the ability to adaptively adjust the output visual marking form according to the target marking intent—that is, generating a box covering the target for "box selection" intents, and generating a flat, narrow box (a box with a height of 0) located at a specified position near the target for "line drawing" intents (covering text and entities), thereby accurately meeting the marking needs of users in different teaching scenarios.

[0040] This specification provides a content tagging method, and also relates to a content tagging model training method, a content tagging device, a content tagging model training device, a computing device, a computer-readable storage medium, and a computer program product, which will be described in detail in the following embodiments.

[0041] See Figure 1 , Figure 1 This specification illustrates a flowchart of a content tagging method according to an embodiment, which specifically includes the following steps: Step 102: Obtain the content to be tagged and the target tagging intent for the content to be tagged.

[0042] It should be noted that the content to be tagged refers to the original information carrier that needs to be tagged, covering multimedia data such as text, images, video, and audio, and can also be specific segments, regions, objects, or semantic units within it. The content to be tagged is the object of the tagging operation. For example, the content to be tagged could be a controversial clause in a legal text, a suspected lesion area in a medical image, an illustration of a specific step in a product instruction manual, or a specific speaker's turn in a meeting recording.

[0043] Target tagging intent refers to the specific semantic purpose that a user expects to achieve when tagging content. This purpose is generally mapped to at least one visual tagging form (such as highlighting, boxing, arrows, coloring, underlining, etc.). Target tagging intent acts as a semantic bridge connecting user needs and model output, enabling content tagging models to shift from "passively executing fixed styles" to "actively understanding tagging requests," thereby generating tag positions and forms that match the intent. Target tagging intent can be natural language instructions (including voice instructions, text instructions, etc.) input into the content tagging model, such as "highlighting revenue peaks" (corresponding to highlighting / circling) in a financial report chart, "indicating the location of screws that need to be tightened" (corresponding to arrows / annotation lines) in an assembly drawing, and "marking blank areas requiring signatures from both parties" (corresponding to boxes / color blocks) in contract text.

[0044] In practical applications, there are various ways to obtain the content to be tagged and the target tagging intent, and the specific method should be selected according to the actual situation. This specification does not impose any limitations on these methods in the embodiments. In one possible implementation of this specification, the content to be tagged and the target tagging intent can be received from the user via a client. In another possible implementation of this specification, the content to be tagged and the target tagging intent can be read from other databases or data acquisition devices.

[0045] Step 104: Input the target labeling intent and the content to be labeled into the content labeling model to obtain the labeling location information. The content labeling model is trained based on the sample labeling content, the sample labeling intent of the sample labeling content, and the sample location information that matches the sample labeling intent. The sample labeling intent corresponds to at least two different sample visual labeling forms.

[0046] It should be noted that the content tagging model is an intelligent annotation model trained based on machine learning. It can receive the content to be tagged and the target tagging intent as input, and output tag position information that matches the intent. Its training data includes sample tagging content, corresponding sample tagging intent, and manually annotated sample position information, and the sample tagging intent is associated with at least two visual tagging forms. For example, the content tagging model is a multimodal large language model.

[0047] The marker location information is computer-readable data describing the precise spatial or temporal location in the content to be marked, where the marker should be applied. Its specific form depends on the content type (e.g., character offsets for text, pixel coordinates / boundary boxes for images, and timestamps + spatial regions for videos). As the instruction carrier between the content marking model and the final rendering module, the marker location information ensures that the marker is accurately attached to the target object, avoiding positional offsets or semantic misalignments. For example, for a product fault image, the marker location information might be [320, 150, 80, 60] (representing a rectangle starting at the top left corner (320, 150), 80 pixels wide, and 60 pixels high); for a contract text, it might be {"start": 245, "end": 260} (indicating that characters 245 to 260 need to be highlighted).

[0048] Visual marker forms are the specific graphic styles or symbol systems used in the visual presentation of markers. They serve as a visual carrier of the marker's intent, and common forms include highlighted blocks, arrows, bounding boxes, underlines, and circles. Visual marker forms transform abstract intents such as "where is important" and "where needs to be operated" into visual language that users can directly perceive. For example, in anatomical atlases, blue arrows point to the direction of blood vessels, and dashed ellipses circle the lymph node groups to be observed. A single image can simultaneously carry multiple visual marker forms, each corresponding to a different marker intent.

[0049] In practical applications, there are various ways to input the target labeling intent and the content to be labeled into the content labeling model to obtain the labeling location information. The specific method is selected according to the actual situation, and the embodiments in this specification do not limit this. In one possible implementation of this specification, the content to be labeled (such as the image to be labeled) and the target labeling intent are concatenated and input into the model. The model directly outputs the labeling location information (such as bounding box coordinates) through cross-modal feature comparison.

[0050] In another possible implementation of this specification, inputting the target tagging intent and the content to be tagged into the content tagging model to obtain tagging location information may include the following steps: Analyze the target's marking intent to obtain the target's visual marking shape; Based on the target visual mark shape, model prompt information is constructed, whereby the model prompt information is used to guide the content markup model to generate mark position information that conforms to the target visual mark shape; Input the model prompts, target labeling intent, and content to be labeled into the content labeling model to obtain labeling location information.

[0051] It should be noted that the target visual markup form refers to the specific visual style category used to present the markup, parsed from the user's input target markup intent. It is one or more combinations of enumerable markup forms such as highlight, arrow, bounding box, underline, color block, and circle annotation. The target visual markup form acts as a mapping bridge from semantic intent to graphical instructions, enabling subsequent model prompt construction and position generation to be accurately aligned around the dimension of "what style to use for the markup," avoiding misalignment between the markup presentation and the user's expectations due to ambiguity in form.

[0052] Model prompts are pre-constructed structured inputs that guide the content labeling model to output location information that conforms to a specific visual marker pattern. These can be natural language instructions, templated descriptions, or example-query pairs, aiming to explicitly tell the model "what visual style is needed for localization in the current task." For example, when the target visual marker pattern is a "red arrow," the model prompt could be "Please point to the target location with the red arrow, and output the coordinates of the arrow's start and end points."

[0053] In practical applications, there are various ways to parse the target marker intent and obtain the target visual marker form. The specific method is selected according to the actual situation, and the embodiments in this specification do not limit this. In one possible implementation of this specification, an "intent-form" mapping dictionary is pre-constructed, and common marker intent keywords are hard-coded and associated with their corresponding visual marker forms. When the target marker intent text is input, the system extracts key intent words through word segmentation, part-of-speech tagging, or regular expression matching, and directly looks up the table to output the preset target visual marker form.

[0054] In another possible implementation of this specification, a pre-trained language model is used to perform deep semantic encoding on the target marked intent text, mapping it to a vector space containing several visual mark morphology categories; after the model is fine-tuned with a small number of labeled samples, it can calculate the probability distribution of the input intent belonging to each type of morphology, or match the closest target visual mark morphology by calculating the cosine similarity between the intent text and the morphology description text.

[0055] In practical applications, there are various ways to construct model prompts based on the target visual marker morphology. The specific method chosen depends on the actual situation, and this specification does not impose any limitations on this approach. In one possible implementation, a fixed-format natural language instruction template is pre-written for each visual marker morphology, and a "morphology-prompt" mapping table is constructed. When the system obtains the target visual marker morphology, it directly calls the corresponding template string as the model prompt information by looking up the table.

[0056] In another possible implementation of this specification, based on the target visual marker shape, several input-output example pairs related to the shape are retrieved from the example library (e.g., if the target shape is a "blue dashed box", then 2 to 3 images previously labeled "blue dashed box" and their corresponding bounding box coordinates are selected) to form model prompt information, guiding the model to imitate the output format of the same shape.

[0057] The solution implemented in this specification locks the visual markup form by explicitly parsing the target markup intent and dynamically constructing model prompts for that form, so that the content markup model receives clear guidance on "what style to use for markup" before generating position information; this significantly improves the matching accuracy between the markup position and the user's expected visual style, and is especially suitable for multi-form mixed annotation, weak morphological semantic association, or zero-sample generalization scenarios.

[0058] Step 106: Based on the marker location information, mark the content to be marked to obtain the content marking result.

[0059] It should be noted that content tagging results refer to the final visual output or structured annotation file generated after the tagging location information and target visual tagging shape are actually applied to the content to be tagged; it can be an image, document, or video frame directly presented to the user, or it can be a machine-readable data format carrying coordinates, shape identifiers, and semantic tags. For example, the content tagging result is a metal flaw detection image with a red arrow pointing to the crack area.

[0060] In practical applications, there are various ways to mark content based on marker location information and obtain the content marking result. The specific method should be selected according to the actual situation, and the embodiments in this specification do not limit this in any way. In one possible implementation of this specification, a computer vision library is called to directly draw graphics on the content to be marked (such as an image to be marked): based on the marker location information output by the model, the corresponding visual marker shape is rendered using library functions. At the same time, style parameters such as color, line width, and transparency can be attached, and finally, a visual image with markers is generated, saved, or displayed, which is the content marking result.

[0061] In another possible implementation of this specification, marking the content to be marked based on the mark location information to obtain the content marking result may include the following steps: Determine the target visual marker shape corresponding to the marker location information; Based on the target visual markup morphology, the content to be marked is marked to obtain the content marking result.

[0062] It should be noted that the marker location information itself does not contain information about the target visual marker shape, but there is a correspondence between the marker location information and the target visual marker shape. The corresponding target visual marker shape can be determined based on the marker location information. For example, if the marker location information consists of the coordinates of three points, the target visual marker shape is a triangle; if the marker location information consists of the coordinates of four points, the target visual marker shape is a quadrilateral.

[0063] In practical applications, there are various ways to determine the target visual marker shape corresponding to the marker position information. The specific method is selected according to the actual situation, and the embodiments in this specification do not limit this. In one possible implementation of this specification, the marker position information is position coordinates. The target visual marker shape can be determined based on the number of position coordinates in the marker position information, and there is a mapping relationship between the number of position coordinates and the target visual marker shape.

[0064] In another possible implementation of this specification, the marker position information is a character offset, including the character indices of the start and end positions of the marker. The target visual marker shape is determined based on the character indices of the start and end positions of the marker. For example, [15, 28] represents the 15th to the 28th character, and the corresponding target visual marker shape can be an underline, a highlighted block, a wavy line, or a strikethrough.

[0065] In another possible implementation of this specification, the marker position information includes first position coordinates, second position coordinates, third position coordinates, and fourth position coordinates; determining the target visual marker shape corresponding to the marker position information may include the following steps: The target visual marker shape corresponding to the marker location information is determined to be a rectangular marker shape; or, The marker position information includes the fifth position coordinate and the sixth position coordinate; determining the target visual shape corresponding to the marker position information includes: determining that the target visual marker shape corresponding to the marker position information is a line marker shape.

[0066] It should be noted that the first to fourth position coordinates refer to four spatial positioning points used to determine the shape of the rectangular marker. These are usually pixel coordinates or relative position values ​​in the image or document coordinate system. These four points can represent the four vertices of the rectangle, or define the boundary of the rectangle in a specific order (such as top left, top right, bottom right, bottom left). The first to fourth position coordinates provide precise shape and position parameters for the rectangular marker, enabling the rendering module to draw a closed quadrilateral (usually an axis-aligned or rotated rectangle) based on the four coordinate points, thereby selecting the target area to be marked. For example, in an industrial inspection image, the first to fourth position coordinates are (120, 80), (200, 80), (200, 150), and (120, 150), respectively, corresponding to drawing a red rectangle that encloses a screw hole.

[0067] The first to fourth position coordinates typically define the four vertices of a rectangle in sequence (e.g., clockwise or counterclockwise), and the points are connected by straight lines to form a closed quadrilateral. Although the coordinates themselves only store position, their implicit geometric relationships (opposite sides parallel, adjacent sides perpendicular, diagonals equal) can be used to validate or correct input. The first to fourth position coordinates directly define the area covered by the rectangle in an image or document. The order of the four vertices indicates whether the rectangle has been rotated.

[0068] The fifth and sixth position coordinates refer to two spatial positioning points used to determine the shape of line markers, typically representing the start and end points of the line, respectively. The coordinate format is adapted to the content type (e.g., (x, y) for a 2D image, (x, y, z) for a 3D model). The fifth and sixth position coordinates provide the start and end positions for line-type markers (arrows, straight lines, and polyline segments within curves), enabling the rendering module to draw linear symbols of a specified style from the start to the end point. For example, on a circuit board assembly drawing, the fifth position coordinate (350, 220) is the arrow's starting point (pointing to annotation text), and the sixth position coordinate (400, 180) is the arrow's ending point (pointing to a capacitor pin).

[0069] The most basic line coordinates consist of two points: a start point and an end point (such as arrows and straight lines). Complex lines (such as polylines and curves) define the path using a sequence of multiple points. Line coordinates do not require closure; the start and end points are connected by straight lines or curves, allowing for free extension, branching, or intersection. Line coordinates implicitly contain directional information (such as arrows, which need to distinguish between start and end points), and the path direction can be expressed through the sequence of points. Coordinate points typically only store key control points; during rendering, interpolation algorithms (such as linear interpolation and spline interpolation) are used to generate continuous lines.

[0070] A rectangular marker is a marker style that uses a rectangle as its visual frame. It can be presented as a solid line frame, a dashed line frame, a color block with semi-transparent fill, or a marker only at the four corners. The rectangle is usually parallel to the coordinate axes, but it can also be rotated by defining four points. Rectangular markers are used to delineate local areas within content and are one of the most commonly used marker styles in object detection, region annotation, and layout analysis.

[0071] Line markers are visual symbols using linear shapes such as line segments, polylines, or arrows. They can be customized with attributes such as arrowheads, colors, line widths, and dashed lines. The simplest line markers are defined by two points: a start point and an end point. More complex shapes (such as curves) can be derived from multiple points. Line markers broadly refer to all path-based markers, including unidirectional arrows, bidirectional arrows, leader lines, flowchart connectors, and hand-drawn trajectories. They are primarily used to indicate direction, establish visual connections (between text descriptions and objectives), and express movement paths or temporal sequences.

[0072] The scheme implemented in this specification establishes an explicit and deterministic mapping rule between the number of coordinates of the marker location information and the shape of the target visual marker, so that the shape determination process is completely free from complex geometric calculations or semantic analysis. The decision can be completed in milliseconds based solely on the number of coordinate points. The extremely simple logic brings zero learning cost and zero runtime overhead, eliminates the risk of shape ambiguity, and provides a unified paradigm for shape expansion.

[0073] In practical applications, there are various ways to mark the content to be marked based on the target visual mark shape, and the specific method should be selected according to the actual situation. This specification does not limit the specific method used in this embodiment. In one possible implementation of this specification, after obtaining the target visual mark shape and the corresponding mark position information (coordinates of four points, start and end points, center radius, etc.), the drawing interface of a graphics library such as OpenCV is directly called to draw the shape on the content to be marked according to preset style parameters (color, line width, line type, transparency). After drawing, the content to be marked is saved as a file or displayed directly to obtain the content marking result.

[0074] In another possible implementation of this specification, the target visual marker shape and marker position information are serialized into a machine-readable intermediate format (such as JSON) and stored decoupled from the original content to be marked; then a dedicated rendering service or front-end component reads the structured file and dynamically draws the shape and coordinates onto the content to be marked to obtain the content marking result.

[0075] The solution implemented in this specification automatically determines the corresponding target visual marker shape from the marker location information, transforming the shape preference that originally required manual recognition or external input into an automated decision driven by coordinate features, and directly completing the marker rendering based on the shape; it realizes a one-stop closed loop from "location data" to "complete visual marker", completely eliminating the efficiency loss and human error risk caused by the manual shape specification step, while enabling the system to flexibly adapt to multiple shape outputs based on the same set of location information, significantly enhancing the expressiveness and scene adaptability of the marking results.

[0076] In one optional embodiment of this specification, marking the content to be marked based on the marking location information to obtain the content marking result may include the following steps: Based on the marker location information, the target marker area in the content to be marked is determined; Perform content recognition on the target marked region to obtain the target marked content; The target content is tagged to obtain the content tagging result.

[0077] It should be noted that the target marking region refers to the specific spatial range or data segment delineated within the content to be marked based on the marking location information. This region is the direct target of subsequent content recognition operations. Its shape is closely related to the content type; in images, it is represented by pixel regions (such as bounding boxes, polygons, and masks); in text, it is represented by character intervals (such as from start offset to end offset); and in audio and video, it is represented by time windows or combined spatial and temporal segments. The target marking region transforms the marking intent from abstract positioning instructions into computable and configurable physical or logical units.

[0078] Target-labeled content refers to a local region or data fragment with actual semantic information or significant features extracted from the target-labeled region using content recognition technology. It can be a sub-region within the target-labeled region (such as the smallest bounding rectangle of an object instance), a key object (such as a face or defect), or a core text unit (such as keywords or entities), representing the "core content" that the user truly intends to label. This deepens the labeling operation from "limiting the scope" to "locking the focus," filtering out background or redundant information within the target-labeled region, allowing subsequent semantic annotation, data storage, or analysis processes to directly impact the most valuable information carrier.

[0079] In practical applications, there are multiple ways to determine the target marking region in the content to be marked based on the marking position information. The specific method should be selected according to the actual situation, and this specification does not limit this method. In one possible implementation of this specification, for image-type content to be marked, when the marking position information is the coordinates of a rectangular bounding box (such as the pixel coordinates of the upper left and lower right corners), the image processing library is directly called to extract the pixel matrix within the coordinate range from the original image array and save it as a new independent image or temporary memory object. This sub-image is the target marking region.

[0080] In another possible implementation of this specification, for text-type content to be marked, when the marking position information is a character-level start and end offset, the native string slicing syntax of the programming language is directly used to extract a continuous character sequence within a specified range from the original text, and the returned substring is the target marking region.

[0081] There are various ways to obtain the target labeled content by performing content recognition on the target labeled region. The specific method is selected according to the actual situation, and the embodiments in this specification do not limit this. In one possible implementation of this specification, the target labeled region image is input into a lightweight pre-trained target detection model. The model performs secondary localization within the region and outputs the bounding box coordinates and category labels of salient objects or defects in the region. The system selects the bounding box with the highest confidence from the detection results as the target labeled content (i.e., "partial region with content"), while retaining its classification semantics.

[0082] In another possible implementation of this specification, for images with obvious contrast between foreground and background within the target marking area (such as handwritten characters, printed symbols, and microscopic cells), image processing techniques (such as threshold segmentation, connected component labeling, and contour finding in OpenCV) are used to automatically extract the bounding rectangle or polygon contour of the independent object; each closed contour is a "partial region with content", which is filtered by area, position, or shape features and used as the target marking content.

[0083] There are multiple ways to mark target content and obtain content marking results, and the specific method chosen depends on the actual situation. This specification does not limit the specific method used in the embodiments. In one possible implementation, the outline of the target content is determined based on the target content, and the outline of the target content is marked to obtain the content marking result.

[0084] In another possible implementation of this specification, the target content is marked based on a preset visual markup shape to obtain the content marking result. For example, the target content is marked based on a rectangle of a preset size.

[0085] The scheme implemented in this specification first locks the target marking area based on the marking location information, then performs content recognition on the area to extract the core semantic content, and finally performs refined marking on the identified content. This upgrades the marking result from "where to mark" to "what to mark", significantly improving the accuracy and information density of the marking. The final content marking result can be rendered as an intuitive multi-level visual mark, or output as structured data containing semantic tags, realizing the intelligence, refinement and value maximization of the marking process.

[0086] In one optional embodiment of this specification, before marking the content to be marked based on the marking location information and obtaining the content marking result, the following steps may also be included: Analyze the target's marking intent to obtain the target's visual marking shape; Based on the marker location information, the content to be marked is marked to obtain the content marking results, including: Based on the location information of the marker and the visual shape of the target marker, the content to be marked is marked to obtain the content marking result.

[0087] It should be noted that there is no direct correspondence between the marker location information and the target visual marker shape. In such cases, it is necessary to analyze the target marker intent to obtain the target visual marker shape. For example, if the marker location information consists of the coordinates of two positions, it is impossible to determine whether the target visual marker shape is a line marker shape or a rectangular marker shape based solely on these coordinates.

[0088] The target marking intent can be an explicit intent that includes the target visual markup form, such as "circle the cat in the picture", in which case the target visual markup form is a rectangle; the target marking intent can also be an implicit intent that does not include the target visual markup form: for example, when marking text content, the target visual markup form is an underline; when marking blank areas, the target visual markup form is a rectangle.

[0089] In practical applications, there are various ways to parse the target marker intent and obtain the target visual marker form. The specific method is selected according to the actual situation, and the embodiments in this specification do not limit this. In one possible implementation of this specification, an intent-form mapping table is pre-constructed, and common marker intent keywords (such as "box out" and "circle out" corresponding to rectangles / ellipses, "pointing" and "indicating" corresponding to arrows, and "highlight" and "emphasize" corresponding to color blocks) are hard-coded and associated with the corresponding visual marker forms. When the target marker intent text is input, the core intent words are extracted through simple string matching (such as regular expressions or keyword search), and the preset target visual marker form is directly output by looking up the table.

[0090] In another possible implementation of this specification, a small amount of intent text data with morphological labels (such as “mark the red element in the picture” → “red arrow”, “emphasize the conclusion sentence” → “yellow highlight”) is collected, a lightweight pre-trained language model is fine-tuned or a simple Naive Bayes classifier is trained; after inputting the target labeled intent text, the model predicts the visual label morphological category to which it belongs, and outputs the morphology with the highest probability as the parsing result.

[0091] There are multiple ways to mark the content to be marked based on the marker location information and the target visual marker shape, and the specific method should be selected according to the actual situation. This specification does not limit the specific method used in the embodiments. In one possible implementation of this specification, the content to be marked is loaded as an image (e.g., using OpenCV). Based on the marker location information (e.g., bounding box coordinates, arrow start and end points, polygon vertices) and the target visual marker shape (rectangle, arrow, ellipse, etc.), the corresponding drawing function is called and style parameters (color, line width, line type) are set to draw the marker directly on the image. After the drawing is completed, the image is saved or displayed to obtain the visualized content marking result.

[0092] In another possible implementation of this specification, the marker location information and the target visual marker shape are stored as a structured file (such as JSON), which is decoupled from the content to be marked. Then, the structured file is read by a front-end component or a dedicated rendering tool, and the marker is dynamically drawn on the content to be marked to obtain the content marking result.

[0093] By applying the solution in the embodiments of this specification, a complete automated link from "understanding intent" to "precise presentation" is constructed by first parsing the target marking intent to lock the visual marking form, and then combining the marking position information to perform marking rendering. This achieves a deep binding between marking style and user semantic needs—the parsing stage ensures that the form selection conforms to the intent (such as "emphasis" corresponding to highlight, "pointing" corresponding to arrow), and the rendering stage integrates the form and position for output, so that the final content marking result is not only accurately located, but also accurately conveys the user's marking purpose at the visual language level.

[0094] In one optional embodiment of this specification, before inputting the target tagging intent and the content to be tagged into the content tagging model to obtain tagging location information, the following steps may also be included: Acquire sample data, which includes sample label content, sample label intent, and sample location information; Input the sample labeling content and sample labeling intent into the content labeling model to obtain predicted location information; Based on the sample location information and the predicted location information, the parameters of the content labeling model are adjusted to obtain the trained content labeling model.

[0095] It should be noted that sample data refers to the original dataset used to train the content labeling model. Each sample data contains three core elements: sample labeling content (the original material to be labeled), sample labeling intent (a description of the user's expected labeling purpose), and sample location information (the precise location of the manually labeled data that matches the intent).

[0096] The sample labeling content refers to the original material to be labeled in the sample data. This can be multimodal data such as images, text paragraphs, video frames, and audio clips, and is the object of the labeling operation. It provides the model with the visual or semantic feature foundation needed for location prediction, from which the model needs to extract key information related to the labeling intent. For example, the sample labeling content could be an industrial quality inspection image containing circuit boards.

[0097] Sample labeling intent is natural language text or structured instructions in sample data that describe the user's intended labeling purpose; it serves as a crucial semantic bridge connecting content and location. Guiding the model to focus on specific regions or objects within the content that are relevant to the intent allows the model to learn and understand "what the user wants to label" rather than blindly detecting all objects. For example, the sample labeling intent might be "Please point the red arrow to the short-circuit capacitor on the circuit board."

[0098] Sample location information refers to manually labeled location data in the sample data that precisely matches the intended sample labeling. It serves as a supervisory signal during model training. As a comparison standard for model prediction results, the difference between the calculated and predicted location information guides the optimization of model parameters, enabling the model to gradually approach the correct localization capability. For example, sample location information includes character offsets [1024, 1093] in text, a list of polygon vertex coordinates [[85, 43], [127, 43], [127, 89], [85, 89]] in images, and timestamps + spatial coordinates [00:13:25, 00:13:28, 320, 240, 80, 60] in videos.

[0099] Content tagging models refer to machine learning models that iterate parameters during training and ultimately learn the ability to map from "content + intent" to "location". Before training, the weights are randomly initialized or pre-trained, and after training, they become intelligent modules capable of automatically predicting tag locations. As the decision engine in the tagging process, it fuses and calculates the sample tagging content and sample tagging intent, and outputs predicted location information that matches the intent.

[0100] Predicted location information, generated during model training, is the real-time location data output by the model after inputting the sample labeled content and sample labeled intent into the current version of the content labeling model. It reflects the model's current capabilities. As an intermediate variable used in loss calculation with the sample location information (true labels), the difference between the predicted location information and the true labels drives model parameter updates through backpropagation, gradually bringing the prediction closer to the true annotations. For example, in one training iteration, the model's predicted output for a circuit board image and the intent of "short-circuit capacitor" is [312, 145, 85, 62], while the manually labeled true sample location information is [320, 150, 80, 60]. The difference between the two is used to calculate the loss and adjust the model weights.

[0101] In practical applications, there are various ways to obtain sample data, and the specific method chosen depends on the actual situation. This specification does not limit the methods used in its embodiments. In one possible implementation, this specification can receive sample tagging content, sample tagging intent, and sample location information sent by the user through a client.

[0102] In another possible implementation of this specification, obtaining sample data may include the following steps: Obtain the sample label content and sample label intent, wherein the sample label content carries the sample label visual label form; The sample markers are located to obtain the sample location information.

[0103] It should be noted that the visual marker morphology of a sample refers to the specific style category of visual markers pre-applied manually within the sample marker content. It is one of the enumerable forms such as highlighted blocks, arrows, bounding boxes, underlines, and circles. It serves as a visual representation of the marker intent and is also the direct analysis object of subsequent location recognition algorithms. As a "style label" for sample markers, it serves two purposes: firstly, to distinguish different marker types for targeted processing; and secondly, to provide morphological supervision signals for training the content marker model, enabling the model to learn to associate specific forms with intent and location. For example, on a car parts diagram, if a red arrow points to a bolt, the sample visual marker morphology is an "arrow."

[0104] Sample markers refer to visually marked graphics or symbols that are pre-applied manually onto sample marker content, possessing a specific visual marking form. They are part of the sample marker content, carrying dual information of user intent and target location, and are typically composed of visual elements such as color, shape, and lines. As an intermediary bridge for automatically obtaining sample location information from "marked raw material"—by recognizing the graphic features of the marker (such as arrow pointing, bounding box vertices, and highlighted area outlines) through image processing or computer vision technology—precise sample location information can be deduced, thus enabling the construction of training data without the need for secondary manual annotation. For example, in a teaching chart, if a lecturer hand-draws a blue circle around an organ, this blue circle is a sample marker.

[0105] In practical applications, there are various ways to identify the location of sample markers and obtain sample location information. The specific method is selected according to the actual situation, and this specification does not limit this approach. In one possible implementation of this specification, for sample markers with distinct color features (such as red arrows, yellow highlights, and blue circles), the sample marker content image is first converted to a color space. The marker region is then separated by setting a color threshold to obtain a binary mask. Subsequently, OpenCV is called to extract the contours of connected components in the mask. The marker shape is determined based on the geometric features of the contour (area, aspect ratio, number of convex hull vertices), and the corresponding sample location information is calculated. For example, for bounding box markers, the coordinates of the smallest bounding rectangle of the contour (such as the coordinates of the four vertices) are taken. For arrow markers, the starting and ending coordinates are determined by combining the contour direction and skeleton extraction.

[0106] In another possible implementation of this specification, for line-type or clearly defined sample markers (such as black solid lines or hand-drawn circles), an edge detection algorithm is first used to extract edge features from the image, and then Hough transform is used to detect lines (for rectangles / arrows) or circles (for circle markers). Hough line detection can return the coordinates of the start and end points of the line segment, and Hough circle detection returns the center and radius of the circle. These geometric parameters directly constitute the sample position information.

[0107] The solution implemented in this specification automatically identifies the location of samples already labeled with manual markings, cleverly transforming readily available visual markers (such as arrows, boxes, and highlights) into structured sample location information. This allows for the batch construction of high-quality training data without any additional manual annotation, reducing the repetitive labor of "applying markers first and then re-annotating the locations" in traditional training data construction and lowering sample collection costs.

[0108] In one optional embodiment of this specification, the sample markers include at least one of the following: underline, circle, square, triangle, strikethrough, wavy line, and arrow.

[0109] It should be noted that an underline is a horizontal line drawn below the text to emphasize or identify specific text content. Its purpose is to draw the user's attention to key sentences, headings, or text passages that require special attention.

[0110] A circle is a closed circular line used to enclose a specific area or object in an image, document, or page. Its purpose is to draw attention to the object within the circle and is often used to annotate details, identify problem areas, or indicate selection.

[0111] A box is a closed rectangle formed by four straight lines, used to select text paragraphs, image areas, or specific objects. Its function is to define a rectangular area to emphasize a specific block on the page, mark detection targets, or serve as a carrier for annotations.

[0112] A triangular frame is a closed triangle formed by three straight lines, used to mark a specific area and imply directionality. Its function is both to define a range and to convey directional clues through the direction of its vertices. It is often used for warnings, identifying key points, or emphasizing areas requiring attention.

[0113] A strikethrough is one or more horizontal lines running through the middle of text, indicating that the text has been deleted, is obsolete, or is no longer applicable. Its purpose is to clearly express the undoing, modification, or completion status of content in document revision, editing, or task management.

[0114] A wavy line is a curved line that resembles a wave, usually drawn below or above text to mark specific passages. Its function depends on the context; it can indicate emphasis, questions, spelling errors, or elegant phrases.

[0115] An arrow is a graphic symbol consisting of a straight line and a triangular arrowhead. It is used to indicate direction, point to a specific target, or express process relationships. Its function is to guide the eye from the annotation or label to a specific object, or to convey logical relationships such as movement, cause and effect, or sequence.

[0116] Sample markings can also be highlighted. Highlighting refers to marking styles that make specific content stand out more in the overall layout by changing the background color, text color, font weight, or adding surrounding embellishments (such as adding a frame or shadow).

[0117] Sample marking can also be symbol marking, which refers to the use of graphic symbols with specific semantics (such as asterisks, question marks, exclamation marks, checkmarks, crosses, etc.) to mark content; these symbols are usually independent of the content itself and are attached to the target location in a superimposed form.

[0118] By incorporating various visual marker forms such as underlines, circles, squares, triangles, strikethroughs, wavy lines, and arrows into the sample marker set, the training data can comprehensively cover diverse marking scenarios such as text emphasis, region delimitation, content deletion and modification, question annotation, and directional indication. This enhances the semantic representation and morphological generalization capabilities of the content marking model for different marking intentions. During the learning process, the model can fully understand the correspondence between various forms and intentions (such as underlines corresponding to emphasis, strikethroughs corresponding to invalidation, and arrows corresponding to direction), thereby automatically matching appropriate visual forms according to user intentions in practical applications.

[0119] In one optional embodiment of this specification, the sample marking intent includes at least one of a sample box selection intent and a sample line drawing intent, wherein the sample box selection intent is used to select a first target region in the sample marking content, and the sample line drawing intent is used to draw a line on a second target region in the sample marking content. The first target region and the second target region can be any region in the sample marking content.

[0120] The sample marker intent is the sample line intent, the second target area is the second object area, and the sample marker position is the line position located at the edge of the second object area. The second object area can be any area containing objects within the sample marker content, such as a line of text.

[0121] The first target region includes a first object region and candidate blank regions. The sample selection intent includes at least one of a sample object selection intent and a sample blank selection intent. The sample object selection intent is used to select the first object region in the sample labeled content, and the sample blank selection intent is used to select the candidate blank regions in the sample labeled content. The first object region can be any region in the sample labeled content where an object exists, and the candidate blank regions can be any blank regions in the sample labeled content.

[0122] The sample selection intent is the sample object selection intent, and the sample marker position is the polygon position covering the first object area.

[0123] The sample selection intent is the sample blank selection intent, the sample marker position is the polygon position covering the target blank area, and the target blank area is divided from the candidate blank area.

[0124] The sample marker location includes the marker shape identifier and the marker coordinates corresponding to the marker shape identifier.

[0125] The scheme implemented in this specification constructs training data consisting of triplet data containing sample labeling content, sample labeling intent, and corresponding sample location information. This drives the content labeling model to perform end-to-end supervised learning, enabling the model to autonomously learn the ability to predict label locations from the joint input of "content + intent". A complete mapping mechanism from semantic understanding to spatial localization is established. The model no longer relies on manually preset rules or templates, but automatically captures the correlation between key clues in intent and relevant features in content through a large amount of sample data, thereby achieving accurate response to diverse and complex labeling intents.

[0126] The scheme described in this specification involves obtaining the content to be labeled and the target labeling intent for the content; inputting the target labeling intent and the content to be labeled into a content labeling model to obtain labeling location information, wherein the content labeling model is trained based on sample labeled content, the sample labeling intent of the sample labeled content, and sample location information matching the sample labeling intent, and the sample labeling intent corresponds to at least two different sample visual labeling forms; based on the labeling location information, the content to be labeled is labeled to obtain the content labeling result. By introducing the target labeling intent and integrating sample training of multi-form visual labels, the content labeling model can accurately understand the user's labeling requirements and automatically generate appropriate labeling location information; it achieves an intelligent upgrade from "single fixed style labeling" to "intent-driven, multi-form adaptation", significantly improving the accuracy and flexibility of labeling, while greatly reducing the operational cost of manually locating and adjusting the labeling form one by one, providing an efficient and unified automated solution for batch and multi-type content labeling tasks.

[0127] The following is in conjunction with the appendix Figure 2 Taking the content tagging method provided in this manual as an example in a teaching scenario, the content tagging method will be further explained. Among other things, Figure 2 A flowchart illustrating the processing procedure of an image region drawing method provided in one embodiment of this specification is shown.

[0128] Step 1: Intent Category Analysis. Researchers randomly select original teaching images (i.e., sample labeled content) from the question bank and construct sample labeled intents based on the answer direction or key information points of the questions. Sample labeled intents can be divided into three categories: box selection / circling, underlining / emphasis, and blank positioning / circling.

[0129] For the selected category, the sample labeling intent could be "circle '50 meters' in the question stem", "circle the triangle in the figure", or "circle option A".

[0130] For the underline category, the sample labeling intent could be "underline the wrong word", "underline the time mentioned in the text", or "underline the rabbit in the picture".

[0131] For blank positioning categories, the sample labeling intent could be "draw a frame to the right of the longer right-angled side of the triangle" or "draw a frame in the first blank area in the question stem".

[0132] Annotators label the target area (or target object) according to the sample labeling intent. For the bounding box category, a regular bounding box that completely covers the target object (whether it's a text block or a visual entity) is labeled to determine the sample location information. For the line category, a flat, narrow bounding box with minimal height (which can be 0) located at the bottom edge of the target object is labeled to determine the sample location information. For the blank positioning category, the target area is defined based on logical judgment, and the corresponding blank area is labeled (i.e., a rectangular sample label) to determine the sample location information. The target area is the target sample labeling area, and the target object is the target sample labeling content.

[0133] Step 2: Format as multimodal JSON. Build the data into the following format: {"system": "As an image coordinate localization expert, it is specifically responsible for accurately locating the specific area in the image that needs to be operated on (such as selecting or drawing) according to user instructions (i.e., sample label content), and outputting the corresponding coordinate values ​​(i.e., predicted location information). Its function is limited to location recognition and coordinate output; it does not perform any actual operation steps."} "conversations": [ { "role": "user", "content": [{"type": "text", "text": "<|image|>\n Command\n Underline "August 6th"}, {"type": "image_path", ...}] }, { "role": "assistant", "content": [{"type": "text", "text": "line:<|BBox|>"}, {"type": "BBox", "BBox": [194, 840, 354, 840]}] } ] }

[0134] In this context, "system" represents the system tag, "conversations" represents the conversation tag, "role" represents the role tag, "user" represents the user, "content" represents the content tag, "type" represents the type tag, "text" represents the text tag, <|image|> represents the image (i.e., the sample labeled content), "image_path" represents the image path, "assistant" represents the assistant, "line" represents the line, <|BBox|> represents the bounding box, and "BBox" represents the bounding box label.

[0135] The "line" (i.e., the marker shape identifier) ​​indicates that [194, 840, 354, 840] represents a line, not a rectangle. [194, 840] represents the coordinates of the first point, and [354, 840] represents the coordinates of the second point.

[0136] Step 3: SFT Training. Use the sample data to perform SFT training on the content tagging model to obtain the trained content tagging model.

[0137] Step 4: Inference and Rendering. The user inputs the image (i.e., the content to be labeled) and the instruction (i.e., the target labeling intent of the content to be labeled) into the content labeling model. The model infers and outputs [x1, y1, x2, y2] (i.e., labeling location information). The front end performs unified rendering based on the labeling location information to obtain the content labeling result. Visually, regular boxes are displayed as selection boxes, and flat narrow boxes are displayed as dashes / underlines.

[0138] This specification proposes a geometric attribute adaptive mapping and data construction method based on intent category-based marker location information. It also proposes a differentiated data annotation strategy by mapping the "box selection" category to a regular rectangular box covering the target marker content and mapping the "line" category to a flat, narrow box located at the bottom of the target marker content.

[0139] This specification proposes a unified localization technology for all target types, and puts forward a localization method that breaks through the dependence on traditional visual features. By constructing target marking intents that include abstract target marking content such as "answer area", "question stem area" and "blackboard area", it establishes a strong correlation between abstract semantic concepts and image spatial coordinates by utilizing the cross-modal alignment capability of a multimodal large language model.

[0140] Compared to general multimodal models that typically output only single-form rectangular bounding boxes and cannot distinguish between "selection" and "underline" categories, the embodiments in this specification achieve semantic adaptation of the target visual marker form by introducing a differentiated annotation strategy based on intent category during the data construction stage. Specifically, the "underline" category is mapped to a flat, narrow box (with minimal height) located at the bottom of the target marker content, and the "selection" intent is mapped to a rectangular box covering the entire target marker content. A mapping relationship between the semantic intent and the geometric attributes of the marker position information is established using SFT. This approach allows the model to adaptively adjust the aspect ratio and relative position of the marker position information based on the implicit interactive intent (such as "circle" or "underline") within the target marker intent.

[0141] To address the issue that related OCR or object detection technologies heavily rely on text pixels or entity visual features, making it unable to locate non-entity / text targets such as "blank answer areas," this specification's embodiments achieve unified and accurate localization for all target types. A target labeling intent is constructed, including targets such as blank areas and abstract text fragments. Utilizing the cross-modal alignment capability of a multimodal large language model, abstract semantic concepts such as "answer area" and "paragraph gaps" are strongly correlated with specific label location information in the image. This allows the model to move beyond simply searching for targets with significant visual features, achieving accurate localization for all target types.

[0142] See Figure 3 , Figure 3 This specification shows a flowchart of a content tagging model training method according to an embodiment, which specifically includes the following steps: Step 302: Obtain sample data, wherein the sample data includes sample label content, sample labeling intent of the sample label content, and sample location information that matches the sample labeling intent, and the sample labeling intent corresponds to at least two different sample visual labeling forms.

[0143] Step 304: Input the sample labeling content and sample labeling intent into the content labeling model to obtain the predicted location information.

[0144] Step 306: Adjust the parameters of the content tagging model based on the sample location information and the predicted location information to obtain the trained content tagging model.

[0145] It should be noted that the implementation methods of steps 302 to 306 are different from those of... Figure 1 The training method for the "content tagging model" in the illustrated content tagging methods is the same, so the embodiments in this specification will not be described again.

[0146] The scheme implemented in this specification constructs training data consisting of triplet data containing sample labeling content, sample labeling intent, and corresponding sample location information. This drives the content labeling model to perform end-to-end supervised learning, enabling the model to autonomously learn the ability to predict label locations from the joint input of "content + intent". A complete mapping mechanism from semantic understanding to spatial localization is established. The model no longer relies on manually preset rules or templates, but automatically captures the correlation between key clues in intent and relevant features in content through a large amount of sample data, thereby achieving accurate response to diverse and complex labeling intents.

[0147] Considering the large number of parameters in the content tagging model and the limited computing resources on the client side, the content tagging method proposed in the embodiments of this specification can be applied to, for example... Figure 4 The content tagging system shown is not limited to this. See also Figure 4 , Figure 4 This specification illustrates an architecture diagram of a content tagging system according to one embodiment. The content tagging system may include a client 402 and a server 404. Client 402 is used to send the content to be tagged and the target tagging intent of the content to be tagged to server 404; The server 404 is used to input the target labeling intent and the content to be labeled into the content labeling model to obtain labeling location information. The content labeling model is trained based on the sample labeling content, the sample labeling intent of the sample labeling content, and the sample location information that matches the sample labeling intent. The sample labeling intent corresponds to at least two different sample visual labeling forms. Based on the labeling location information, the content to be labeled is labeled to obtain the content labeling result. The content labeling result is sent to the client 402. Client 402 is also used to receive the content tagging result sent by server 404.

[0148] like Figure 4As shown, the content tagging model is deployed in server 404. Server 404 can connect to one or more clients 402 via a local area network (LAN), wide area network (WAN), Internet, or other types of data network. Data transmitted by client 402 may require encoding, transcoding, compression, or other processing before being published to server 404. Client 402 can also interact with users through a graphical user interface to invoke the content tagging model, thereby implementing the content tagging method provided in this embodiment. Multiple clients 402 can establish communication connections through server 404. In the content tagging scenario, server 404 provides content tagging services between multiple clients 402. Multiple clients 402 can act as senders or receivers, communicating through server 404. Users can interact with server 404 through client 402 to receive data sent by other clients 402, or send data to other clients 402, etc. In a content tagging scenario, a user can publish the content to be tagged and the target tagging intent of the content to be tagged to the server 404 through the client 402. The server 404 generates the content tagging result based on the content to be tagged and the target tagging intent of the content to be tagged, and pushes the content tagging result to other clients that have established communication.

[0149] Client 402 can be a browser, application (APP), or web application such as HyperText Markup Language 5 (H5) application, or a lightweight application (also known as a mini-program), or cloud application, etc. Client 402 can be developed based on the software development kit (SDK) of the corresponding service provided by server 404, such as based on the Real-Time Communication (RTC) SDK. Client 402 can be deployed in electronic devices and depends on the device to run or on certain APPs on the device. Electronic devices may have a display screen and support information browsing, such as personal mobile terminals such as mobile phones, tablets, and personal computers (PCs). Various other types of applications can also be configured in electronic devices, such as human-computer interaction applications, model training applications, text processing applications, web browser applications, shopping applications, search applications, instant messaging tools, email clients, social platform software, etc.

[0150] A server-side 404 error can include servers providing various services, such as servers providing communication services to multiple clients, servers supporting backend training of models used on clients, and servers processing data sent by clients. It's important to note that a server-side 404 error can be implemented as a distributed server cluster composed of multiple servers, or as a single server. The server can also be a server in a distributed system, or a server integrated with blockchain. Furthermore, the server can be a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms, or an intelligent cloud computing server or intelligent cloud host with artificial intelligence technology.

[0151] It is worth noting that the content tagging methods provided in the embodiments of this specification are generally executed by the server. However, in other embodiments of this specification, if the client's runtime resources can meet the deployment and runtime conditions of the content tagging model, the client can also have similar functionality to the server, thereby executing the content tagging methods provided in the embodiments of this specification. In other embodiments, the content tagging methods provided in the embodiments of this specification can also be executed jointly by the client and the server.

[0152] Corresponding to the above method embodiments, this specification also provides embodiments of content marking devices. Figure 5 A schematic diagram of a content marking device according to one embodiment of this specification is shown. Figure 5 As shown, the device includes: The first acquisition module 502 is configured to acquire the content to be labeled and the target labeling intent of the content to be labeled. The first input module 504 is configured to input the target labeling intent and the content to be labeled into the content labeling model to obtain labeling location information. The content labeling model is trained based on the sample labeling content, the sample labeling intent of the sample labeling content, and the sample location information that matches the sample labeling intent. The sample labeling intent corresponds to at least two different sample visual labeling forms. The tagging module 506 is configured to tag the content to be tagged based on the tagging location information to obtain the content tagging result.

[0153] Optionally, the first input module 504 is further configured to parse the target marking intent to obtain the target visual marking shape; construct model prompt information based on the target visual marking shape, wherein the model prompt information is used to guide the content marking model to generate marking position information that conforms to the target visual marking shape; and input the model prompt information, the target marking intent, and the content to be marked into the content marking model to obtain the marking position information.

[0154] Optionally, the marking module 506 is further configured to determine the target visual marking shape corresponding to the marking position information; and to mark the content to be marked based on the target visual marking shape to obtain the content marking result.

[0155] Optionally, the marker position information includes a first position coordinate, a second position coordinate, a third position coordinate, and a fourth position coordinate; the marker module 506 is further configured to determine that the target visual marker shape corresponding to the marker position information is a rectangular marker shape; the marker position information includes a fifth position coordinate and a sixth position coordinate; the marker module 506 is further configured to determine that the target visual marker shape corresponding to the marker position information is a line marker shape.

[0156] Optionally, the marking module 506 is further configured to determine the target marking region in the content to be marked based on the marking location information; perform content recognition on the target marking region to obtain the target marking content; and mark the target marking content to obtain the content marking result.

[0157] Optionally, the tagging module 506 is also configured to parse the target tagging intent, obtain the target visual tagging shape, and tag the content to be tagged based on the tagging location information and the target visual tagging shape to obtain the content tagging result.

[0158] Optionally, the device further includes: a training module; the training module is configured to acquire sample data, wherein the sample data includes sample labeling content, sample labeling intent, and sample location information; input the sample labeling content and sample labeling intent into the content labeling model to obtain predicted location information; and adjust the parameters of the content labeling model according to the sample location information and predicted location information to obtain a trained content labeling model.

[0159] Optionally, the training module is also configured to acquire sample label content and sample label intent, wherein the sample label content carries sample labels with sample visual label morphology; perform position recognition on the sample labels to obtain sample position information; the sample labels include at least one of underline, circle, square, triangle, strikethrough, wavy line, and arrow.

[0160] The solution implemented in this specification, by introducing target marking intent and integrating sample training with multi-form visual marking, enables the content marking model to accurately understand user marking requirements and automatically generate appropriate marking location information. This achieves an intelligent upgrade from "single fixed-style marking" to "intent-driven, multi-form adaptation," significantly improving the accuracy and flexibility of marking, while greatly reducing the operational cost of manually locating and adjusting marking forms one by one. It provides an efficient and unified automated solution for batch and multi-type content marking tasks.

[0161] The above is an illustrative scheme of a content marking device according to this embodiment. It should be noted that the technical solution of this content marking device and the technical solution of the above-described content marking method belong to the same concept. For details not described in detail in the technical solution of the content marking device, please refer to the description of the technical solution of the above-described content marking method.

[0162] Corresponding to the above method embodiments, this specification also provides embodiments of a content tagging model training device. Figure 6 A schematic diagram of a content tagging model training device according to one embodiment of this specification is shown. Figure 6 As shown, the device includes: The second acquisition module 602 is configured to acquire sample data, wherein the sample data includes sample marking content, sample marking intent of the sample marking content, and sample location information matching the sample marking intent, and the sample marking intent corresponds to at least two different sample visual marking forms. The second input module 604 is configured to input the sample labeling content and the sample labeling intent into the content labeling model to obtain the predicted location information; The adjustment module 606 is configured to adjust the parameters of the content labeling model based on the sample location information and the predicted location information to obtain the trained content labeling model.

[0163] The scheme implemented in this specification constructs training data consisting of triplet data containing sample labeling content, sample labeling intent, and corresponding sample location information. This drives the content labeling model to perform end-to-end supervised learning, enabling the model to autonomously learn the ability to predict label locations from the joint input of "content + intent". A complete mapping mechanism from semantic understanding to spatial localization is established. The model no longer relies on manually preset rules or templates, but automatically captures the correlation between key clues in intent and relevant features in content through a large amount of sample data, thereby achieving accurate response to diverse and complex labeling intents.

[0164] The above is a schematic scheme of a content tagging model training device according to this embodiment. It should be noted that the technical solution of this content tagging model training device and the technical solution of the content tagging model training method described above belong to the same concept. For details not described in detail in the technical solution of the content tagging model training device, please refer to the description of the technical solution of the content tagging model training method described above.

[0165] Figure 7 A structural block diagram of a computing device according to one embodiment of this specification is shown. The components of the computing device 700 include, but are not limited to, a memory 710 and a processor 720. The processor 720 is connected to the memory 710 via a bus 730, and a database 750 is used to store data.

[0166] The computing device 700 also includes an access device 740, which enables the computing device 700 to communicate via one or more networks 760. Examples of these networks include Public Switched Telephone Network (PSTN), Local Area Network (LAN), Wide Area Network (WAN), Personal Area Network (PAN), or combinations of communication networks such as the Internet. The access device 740 may include one or more of any type of wired or wireless network interface (e.g., Network Interface Card (NIC)), such as an IEEE 802.11 Wireless Local Area Networks (WLAN) interface, a Wi-MAX (World Interoperability for Microwave Access) interface, an Ethernet interface, a Universal Serial Bus (USB) interface, a cellular network interface, a Bluetooth interface, a Near Field Communication (NFC) interface, and so on.

[0167] In one embodiment of this specification, the above-described components of the computing device 700 and Figure 7 Other components, not shown, can also be connected to each other, for example, via a bus. It should be understood that... Figure 7 The block diagram of the computing device shown is for illustrative purposes only and is not intended to limit the scope of this specification. Those skilled in the art can add or replace other components as needed.

[0168] The computing device 700 can be any type of stationary or mobile computing device, including mobile computers or mobile computing devices (e.g., tablet computers, personal digital assistants, laptop computers, notebook computers, netbooks, etc.), mobile phones (e.g., smartphones), wearable computing devices (e.g., smartwatches, smart glasses, etc.) or other types of mobile devices, or stationary computing devices such as desktop computers or personal computers (PCs). The computing device 700 can also be a mobile or stationary server.

[0169] The processor 720 is used to execute computer programs / instructions, which, when executed by the processor, implement the steps of the content tagging method or content tagging model training method described above.

[0170] The above is an illustrative scheme of a computing device according to this embodiment. It should be noted that the technical solution of this computing device belongs to the same concept as the technical solutions of the content tagging method and the content tagging model training method described above. For details not described in detail in the technical solution of the computing device, please refer to the description of the technical solutions of the content tagging method or the content tagging model training method described above.

[0171] An embodiment of this specification also provides a computer-readable storage medium storing a computer program / instructions that, when executed by a processor, implement the steps of the above-described content tagging method or content tagging model training method.

[0172] The above is an illustrative scheme of a computer-readable storage medium according to this embodiment. It should be noted that the technical solution of this storage medium belongs to the same concept as the technical solutions of the content tagging method and the content tagging model training method described above. For details not described in detail in the technical solution of the storage medium, please refer to the description of the technical solutions of the content tagging method or the content tagging model training method described above.

[0173] An embodiment of this specification also provides a computer program product, including a computer program / instructions that, when executed by a processor, implement the steps of the above-described content tagging method or content tagging model training method.

[0174] The above is an illustrative scheme of a computer program product according to this embodiment. It should be noted that the technical solution of this computer program product belongs to the same concept as the technical solutions of the content tagging method and the content tagging model training method described above. For details not described in detail in the technical solution of the computer program product, please refer to the description of the technical solutions of the content tagging method or the content tagging model training method described above.

[0175] The foregoing has described specific embodiments of this specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims may be performed in a different order than that shown in the embodiments and may still achieve the desired result. Furthermore, the processes depicted in the drawings do not necessarily require the specific or sequential order shown to achieve the desired result. In some embodiments, multitasking and parallel processing are possible or may be advantageous.

[0176] Computer instructions include computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. Computer-readable media can include: any entity or device capable of carrying computer program code, recording media, USB flash drives, portable hard drives, magnetic disks, optical disks, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc. It should be noted that the content included in computer-readable media can be appropriately added or removed according to the requirements of patent practice. For example, in some regions, according to patent practice, computer-readable media do not include electrical carrier signals and telecommunication signals.

[0177] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that the embodiments in this specification are not limited to the described order of actions, because according to the embodiments in this specification, some steps can be performed in other orders or simultaneously. Furthermore, those skilled in the art should also understand that the embodiments described in this specification are all preferred embodiments, and the actions and modules involved are not necessarily essential to the embodiments in this specification.

[0178] In the above embodiments, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0179] The preferred embodiments disclosed above are merely illustrative of this specification. The optional embodiments do not exhaustively describe all details, nor do they limit the invention to the specific implementations described. Clearly, many modifications and variations can be made based on the embodiments described herein. These embodiments are selected and specifically described in this specification to better explain the principles and practical applications of the embodiments, thereby enabling those skilled in the art to better understand and utilize this specification. This specification is limited only by the claims and their full scope and equivalents.

Claims

1. A content tagging method, characterized in that, include: Obtain the content to be labeled and the target labeling intent for the content to be labeled; The target labeling intent and the content to be labeled are input into the content labeling model to obtain labeling location information. The content labeling model is trained based on sample labeling content, the sample labeling intent of the sample labeling content, and sample location information that matches the sample labeling intent. The sample labeling intent corresponds to at least two different sample visual labeling forms. Based on the marked location information, the content to be marked is marked to obtain the content marking result.

2. The method according to claim 1, characterized in that, The step of inputting the target labeling intent and the content to be labeled into the content labeling model to obtain labeling location information includes: The target marking intent is analyzed to obtain the target visual marking shape; Based on the target visual mark shape, model prompt information is constructed, wherein the model prompt information is used to guide the content mark model to generate mark position information that conforms to the target visual mark shape; The model prompt information, the target marking intent, and the content to be marked are input into the content marking model to obtain the marking location information.

3. The method according to claim 1, characterized in that, The step of marking the content to be marked based on the marked location information to obtain the content marking result includes: Determine the target visual marker shape corresponding to the marker location information; Based on the target visual mark shape, the content to be marked is marked to obtain the content marking result.

4. The method according to claim 3, characterized in that, The marker position information includes first position coordinates, second position coordinates, third position coordinates, and fourth position coordinates; determining the target visual marker shape corresponding to the marker position information includes: determining that the target visual marker shape corresponding to the marker position information is a rectangular marker shape; or, The marker position information includes fifth position coordinates and sixth position coordinates; determining the target visual shape corresponding to the marker position information includes: determining that the target visual marker shape corresponding to the marker position information is a line marker shape.

5. The method according to claim 1, characterized in that, The step of marking the content to be marked based on the marked location information to obtain the content marking result includes: Based on the marked location information, the target marked area in the content to be marked is determined; Content recognition is performed on the target marked region to obtain the target marked content; The target content is tagged to obtain the content tagging result.

6. The method according to any one of claims 1 to 5, characterized in that, Before marking the content to be marked based on the marked location information and obtaining the content marking result, the method further includes: The target marking intent is analyzed to obtain the target visual marking shape; The step of marking the content to be marked based on the marked location information to obtain the content marking result includes: Based on the marker location information and the target visual marker shape, the content to be marked is marked to obtain the content marking result.

7. The method according to any one of claims 1 to 5, characterized in that, Before inputting the target labeling intent and the content to be labeled into the content labeling model to obtain labeling location information, the method further includes: Acquire sample data, wherein the sample data includes the sample label content, the sample label intent, and the sample location information; Input the sample labeling content and the sample labeling intent into the content labeling model to obtain predicted location information; Based on the sample location information and the predicted location information, the parameters of the content tagging model are adjusted to obtain the trained content tagging model.

8. The method according to claim 7, characterized in that, The acquisition of sample data includes: Obtain the sample label content and the sample label intent, wherein the sample label content carries the sample label with the visual label form of the sample; The sample markers are used to identify their locations, thereby obtaining the sample location information.

9. The method according to claim 8, characterized in that, The sample markers include at least one of the following: underline, circle, square, triangle, strikethrough, wavy line, and arrow.

10. A content tagging model training method, characterized in that, include: Acquire sample data, wherein the sample data includes sample labeling content, sample labeling intent of the sample labeling content, and sample location information matching the sample labeling intent, wherein the sample labeling intent corresponds to at least two different sample visual labeling forms; Input the sample labeling content and the sample labeling intent into the content labeling model to obtain predicted location information; Based on the sample location information and the predicted location information, the parameters of the content tagging model are adjusted to obtain the trained content tagging model.

11. A computing device, characterized in that, include: Memory and processor; The memory is used to store computer programs / instructions, and the processor is used to execute the computer programs / instructions, which, when executed by the processor, implement the steps of the method according to any one of claims 1 to 10.

12. A computer-readable storage medium, characterized in that, It stores a computer program / instructions that, when executed by a processor, implement the steps of the method according to any one of claims 1 to 10.

13. A computer program product, characterized in that, Includes a computer program / instructions that, when executed by a processor, implement the steps of the method according to any one of claims 1 to 10.

Images