Visual task processing method and related equipment

Abstract

The embodiment of the invention provides a visual task processing method and related equipment, and the method comprises the steps: firstly, inputting an image and a visual task prompt text into a multi-modal language model for visual task processing, and obtaining an output lexical element sequence composed of a target text lexical element and a target coordinate lexical element; next, decoding all the target coordinate lexical elements to obtain target pixel coordinates corresponding to the visual task prompt text in the image; and finally, based on the target pixel coordinates, obtaining a target output result corresponding to the visual task prompt text, and converting the visual task into a multi-modal language model prediction task, thereby significantly improving the accuracy and reliability of performing a visual perception task by using a natural language.

Description

Technical Field

[0001] This application relates to the field of large model data processing technology, and in particular to visual task processing methods and related equipment. Background Technology

[0002] Data processing for visual perception tasks using natural language refers to processing image and natural language query input data into a model and transforming the visual localization task into predicted text information. In related technologies, this data processing flow is primarily based on coordinate regression detection models, such as open-set detection models evolved from YOLO to GroundingDINO. These methods achieve target localization by directly predicting continuous coordinate offsets and outputting the target location as text.

[0003] However, while traditional coordinate regression models are accurate in localization, their language understanding ability is relatively shallow, making it difficult to handle natural language instructions with complex or subtle semantic differences. As a result, they generally face a series of technical problems such as low recall, coordinate drift, and repeated predictions, which in turn leads to low accuracy of prediction results obtained from existing tasks that use natural language for visual perception. Summary of the Invention

[0004] This application provides a visual task processing method and related equipment, which can improve the accuracy of prediction results obtained by processing visual perception task data using natural language.

[0005] To achieve the above objectives, a first aspect of this application provides a visual task processing method, the method comprising: The image and visual task prompt text are input into a multimodal language model for visual task processing, resulting in an output word sequence consisting of target text words and target coordinate words. Decode all the target coordinate words to obtain the target pixel coordinates in the image that correspond to the visual task prompt text; Based on the target pixel coordinates, the target output result corresponding to the visual task prompt text is obtained.

[0006] In some embodiments, the image is input into the visual encoder for visual feature extraction to obtain image features, and the visual task prompt text is input into the text encoder for visual task prompt text word extraction to obtain visual task prompt text words. The image features and the visual task prompt text lexical units are input into the multimodal language model, and the target text lexical units and the output lexical unit sequence are generated using the autoregressive mechanism of the multimodal language model; The vocabulary of the multimodal language model stores the coordinate encoding mapping relationship between each pixel coordinate in the image and a preset number of special coordinate words.

[0007] In some embodiments, the multimodal language model includes an image preprocessing module, which generates the coordinate encoding mapping relationship, including: Based on the size of the image, the coordinates of each pixel in the image are normalized to obtain the corresponding normalized pixel coordinates; Based on a preset number of discrete integers, each normalized pixel coordinate is discretized to obtain a corresponding coordinate quantization value, and each coordinate quantization value corresponds to an image coordinate word. The normalized pixel coordinates are mapped to the image coordinate terms, and the image coordinate terms are added to the vocabulary of the multimodal language model to generate the coordinate encoding mapping relationship. The sequence of two image coordinate terms can be used to characterize a normalized pixel coordinate.

[0008] In some embodiments, decoding all the target coordinate words to obtain the target pixel coordinates in the image corresponding to the visual task prompt text includes: Based on the vocabulary, the target coordinate lexical is converted into the corresponding coordinate quantization value; Based on the coordinate encoding mapping relationship, the coordinate quantization value is restored to the target pixel coordinate corresponding to the target coordinate word, and the target pixel coordinate represents the target object region corresponding to the visual task prompt text.

[0009] In some embodiments, the target text lexicon is used to represent the target text of a visual task, and the target coordinate lexicon is used to represent the location of the target object region corresponding to the target text.

[0010] In some embodiments, when the visual task is general object detection, the target text word is a category phrase word, and the target coordinate word corresponds to the category phrase word.

[0011] In some embodiments, when the visual task is object referential target detection, the target text lexical is a natural language referential sentence lexical, and the target coordinate lexical corresponds to the referential target.

[0012] In some embodiments, when the visual task is visual cue target detection, the target text word is a visual cue sentence word, and the target coordinate word corresponds to the cue target.

[0013] In some embodiments, when the visual task is text recognition, the target text word is the recognized text word, and the target coordinate word is the coordinate word corresponding to the text region.

[0014] In some embodiments, the training steps of the multimodal language model include: Multiple multimodal data samples are acquired, each of which includes image sample data and corresponding label text information and label lexical coordinates; The multimodal data samples are input into the initial multimodal language model, and the first training prediction word sequence is generated using the autoregressive mechanism of the initial multimodal language model. Based on the first training predicted word sequence, the label text information, and the label word coordinates, the cross-entropy loss function is calculated, and the initial multimodal language model is subjected to supervised fine-tuning training to obtain the fine-tuned multimodal language model. At least one sampled training data is obtained from the plurality of multimodal data samples, the sampled training data is input into the fine-tuned multimodal language model, and a second training prediction word sequence is generated using the autoregressive mechanism of the fine-tuned multimodal language model. Based on the second training prediction word sequence, the label text information, and the label word coordinates, a corresponding set of training reward values ​​is calculated. The fine-tuned multimodal language model is trained using reinforcement learning based on the set of training reward values ​​corresponding to the training visual task, thereby obtaining the multimodal language model.

[0015] In some embodiments, when the visual task is an object detection task, the step of acquiring the multimodal data samples includes: Multiple public image data samples are acquired, and the multiple public image data samples are filtered based on a preset resolution and preset compliance constraints to obtain compliant image data samples; Each compliant image data sample is input into the initial multimodal language model to obtain the text description information corresponding to each compliant image data sample; Extract at least one basic category name from each of the text description information, and input each of the compliant image data samples and all the corresponding basic category names into the target detection model for data detection, to obtain the basic detection result of the basic category name in each of the compliant image data samples; The basic detection results are converted to obtain the corresponding tag word coordinates; Based on the compliant image data samples, the basic category names, and the tag lexical coordinates, the multimodal data samples for the target detection task are obtained.

[0016] In some embodiments, when the visual task is a referential object task, the step of acquiring the multimodal sample data includes: Acquire initial multimodal sample data and input the initial multimodal sample data into the initial multimodal language model to obtain corresponding referential language text information. The initial multimodal sample data is image sample data including basic bounding boxes and category labels. The textual information of the referential language and the corresponding image sample data are input into a preset target referential model for data processing to obtain the coordinates of the target referential object. Generate object segmentation masks for all objects in the image sample data, and perform test matching between the coordinate points of the target referencing object and the bounding boxes corresponding to multiple object segmentation masks to obtain test matching results; When the test matching result indicates that the coordinates of the target referent are in the target object segmentation mask, the association information between the coordinates of the target referent and the target object segmentation mask is generated, wherein the target object segmentation mask is one of a plurality of object segmentation masks; Based on the image sample data, the referential language text information, and the bounding box corresponding to the target object segmentation mask, the multimodal data sample of the referential object task is obtained.

[0017] In some embodiments, the step of inputting the multimodal data samples into an initial multimodal language model and generating a first training prediction word sequence using the autoregressive mechanism of the initial multimodal language model includes: Category names are obtained from multiple multimodal data samples, and a global phrase pool is obtained based on all the category names; For each of the multimodal data samples, at least one category name that does not exist in the image sample data is selected from the global phrase pool as a negative query sample; The multimodal data samples and the negative query samples are input into the initial multimodal language model, and the first training prediction word sequence is generated using the autoregressive mechanism of the initial multimodal language model. The first training prediction word sequence corresponding to the negative query sample is a specific sequence of "coordinates not found".

[0018] In some embodiments, calculating a set of training reward values ​​based on the second training predicted lexical sequence, the label text information, and the label lexical coordinates includes: When the second training prediction word sequence is the corresponding target detection box prediction task information, the predicted target detection box is calculated based on the second training prediction word sequence. Based on the intersection-union ratio between the predicted target detection box and the label bounding box corresponding to the label lexical coordinates, a set of training reward values ​​corresponding to the second training predicted lexical sequence is obtained; When the second training prediction word sequence is the corresponding target point prediction task information, and the second training prediction word sequence points to an irregular target object, a sample object segmentation mask is generated based on the irregular target object in the corresponding image data sample, and a set of training reward values ​​corresponding to the second training prediction word sequence is obtained based on the matching result between the predicted target point in the second training prediction word sequence and the bounding box corresponding to the sample object segmentation mask. When the second training prediction lexical sequence is the corresponding target point prediction task information, and the second training prediction lexical sequence points to a regular target object, the object bounding box of the regular target object in the corresponding image data sample is obtained, and a set of training reward values ​​corresponding to the second training prediction lexical sequence is obtained based on the matching result of the predicted target point and the object bounding box of the regular target object.

[0019] In some embodiments, the step of training the fine-tuned multimodal language model using reinforcement learning based on the set of training reward values ​​corresponding to the training visual task to obtain the multimodal language model includes: The set of training reward values ​​is standardized to obtain the relative advantage value within the group; Based on the relative advantage value, the fine-tuned multimodal language model is trained using the policy gradient method with KL divergence constraints to obtain the multimodal language model.

[0020] To achieve the above objectives, a second aspect of this application provides a visual task processing apparatus, the apparatus comprising: The prediction module is used to input images and visual task prompt text into a multimodal language model for visual task processing, and to obtain an output word sequence consisting of target text words and target coordinate words; The pixel coordinate conversion module is used to decode all the target coordinate words to obtain the target pixel coordinates in the image that correspond to the visual task prompt text. The result output module is used to obtain the target output result corresponding to the visual task prompt text based on the target pixel coordinates.

[0021] To achieve the above objectives, a third aspect of this application provides an electronic device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the visual task processing method as described in the first aspect.

[0022] To achieve the above objectives, a fourth aspect of the present application provides a storage medium, which is a computer-readable storage medium storing a computer program that, when executed by a processor, implements the visual task processing method described in the first aspect.

[0023] The visual task processing method and related equipment proposed in this application include: first, inputting an image and visual task prompt text into a multimodal language model for visual task processing to obtain an output word sequence composed of target text words and target coordinate words; then, decoding all target coordinate words to obtain the target pixel coordinates in the image corresponding to the visual task prompt text; and finally, obtaining the target output result corresponding to the visual task prompt text based on the target pixel coordinates. This application addresses the technical problems of low localization accuracy and poor reliability caused by the insufficient language understanding ability of traditional regression models by unifying the visual task into a multimodal language model prediction of sequences containing target text terms and target coordinate terms. The method of this application not only makes full use of the powerful and complex semantic understanding ability of large-scale language models, but also transforms the difficult continuous regression task into a classification task that is easier for the model to learn by discretizing the coordinates, thereby reducing the learning difficulty. At the same time, it provides a unified interface for all visual perception tasks, eliminating the need to design a dedicated detection head. Furthermore, through efficient coordinate term representation, it significantly reduces the length of the generated sequence and improves the inference speed in dense scenes. Ultimately, it effectively solves the problems of high difficulty and easy drift in coordinate regression in the prior art, and significantly improves the accuracy and reliability of visual perception tasks using natural language.

[0024] Other features and advantages of this application will be set forth in the description which follows, and will be apparent in part from the description, or may be learned by practicing the application. The objectives and other advantages of this application may be realized and obtained by means of the structures particularly pointed out in the description, claims and drawings. Attached Figure Description

[0025] Figure 1 This is a flowchart of a visual task processing method provided in an embodiment of this application.

[0026] Figure 2 yes Figure 1 The flowchart for step 101.

[0027] Figure 3 This is a flowchart illustrating the generation of coordinate encoding mapping relationships provided in another embodiment of this application.

[0028] Figure 4 This is a schematic diagram of a multimodal language model performing data autoregression processing, provided in another embodiment of this application.

[0029] Figure 5 This is a schematic diagram of a training architecture based on a multimodal language model provided in one embodiment of this application.

[0030] Figure 6 This is a flowchart of the training process for a multimodal language model provided in another embodiment of this application.

[0031] Figure 7 This is a schematic diagram of the data autoregression processing during the training process of a multimodal language model provided in another embodiment of this application.

[0032] Figure 8 This is a flowchart of the acquisition of multimodal data samples when the vision task is object detection, provided in another embodiment of this application.

[0033] Figure 9 This is a flowchart of the acquisition of multimodal sample data when the vision task is a referential object task, provided in another embodiment of this application.

[0034] Figure 10 yes Figure 6 The flowchart for step 602.

[0035] Figure 11 yes Figure 6 The flowchart for step 605.

[0036] Figure 12 yes Figure 6 The flowchart for step 606.

[0037] Figure 13 yes Figure 1 The flowchart for step 102.

[0038] Figure 14 This is a schematic diagram of the structure of a visual task processing device provided in an embodiment of this application.

[0039] Figure 15 This is a schematic diagram of the hardware structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0040] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0041] It should be noted that although functional modules are divided in the device schematic diagram and the logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the device or the order in the flowchart.

[0042] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit this application.

[0043] Data processing for visual perception tasks using natural language refers to processing image and natural language query input data into a model and transforming the visual localization task into predicted text information. In related technologies, this data processing flow is primarily based on coordinate regression detection models, such as open-set detection models evolved from YOLO to GroundingDINO. These methods achieve target localization by directly predicting continuous coordinate offsets and outputting the target location as text.

[0044] However, while traditional coordinate regression models are accurate in localization, their language understanding ability is relatively shallow, making it difficult to handle natural language instructions with complex or subtle semantic differences. As a result, they generally face a series of technical problems such as low recall, coordinate drift, and repeated predictions, which in turn leads to low accuracy of prediction results obtained from existing tasks that use natural language for visual perception.

[0045] The visual task processing method and related equipment provided in this application will be further described below. First, the visual task processing method in the embodiments of this application will be described in detail. Refer to Figure 1 This is an optional flowchart of the visual task processing method provided in the embodiments of this application. Figure 1 The method described may include, but is not limited to, steps 101 to 103. It is also understood that this embodiment... Figure 1 The order of steps 101 to 103 is not specifically limited; the order of steps can be adjusted or certain steps can be added or removed according to actual needs. The visual task processing method provided in this application can be applied to any control system with computing resources (such as servers, control processors, smart terminals, etc.).

[0046] Step 101: Input the image and visual task prompt text into the multimodal language model for visual task processing to obtain the output word sequence consisting of target text words and target coordinate words.

[0047] Step 101 will be described in detail below.

[0048] In some embodiments, when the control system responds to a data processing request for a visual task, it first acquires the image and visual task prompt text input in response to the visual task. In this step, the image is unstructured data carrying visual information, such as a still image or video frame; the visual task prompt text is a natural language instruction input by the user to describe a specific visual task, such as "detect the cat in the image" or "the man in the blue jacket on the far left." By receiving data from these two different modalities, necessary input information is provided for subsequent processing by a multimodal language model, enabling the execution of the visual task to be precisely aligned with the user's natural language intent.

[0049] In some embodiments, the image and visual task prompt text are then input into a multimodal language model for data processing to obtain target text lexical units and an output lexical sequence consisting of at least one target coordinate lexical unit. This multimodal language model is an artificial intelligence model capable of collaboratively understanding image and text information. The model's data processing transforms the visual localization problem into a unified sequence generation task. In its output, target text lexical units are natural language words describing the identified object (e.g., "person"), while target coordinate lexical units are special technical features that refer to special symbols predefined in the model's vocabulary that form a one-to-one correspondence with quantized discrete coordinate values ​​on the image. In this way, the model can seamlessly output structured information containing "what" (target text lexical units) and "where" (target coordinate lexical units) in a language sequence generation mode.

[0050] In this application, the same autoregressive mechanism used to generate natural language sentences is employed to predict the next output terminology. Upon receiving the image and text context, the model predicts the next most likely terminology one by one. For example, for the query text "detect people and seagulls," the model's output is a coherent text stream that naturally combines descriptive text with predicted coordinate information.

[0051] The following section will further describe the process of inputting image data and visual task prompt text into a multimodal language model for data processing.

[0052] Reference Figure 2 The image and visual task prompt text are input into a multimodal language model for visual task processing to obtain an output word sequence consisting of target text words and target coordinate words, including the following steps 201 to 202.

[0053] Step 201: Input the image into the visual encoder for visual feature extraction to obtain image features, and input the visual task prompt text into the text encoder for visual task prompt text word extraction to obtain visual task prompt text words.

[0054] Step 202: Input the image features and visual task prompt text lexical units into the multimodal language model, and use the autoregressive mechanism of the multimodal language model to generate target text lexical units and output lexical unit sequences; wherein, the vocabulary of the multimodal language model stores the coordinate encoding mapping relationship between each pixel coordinate in the image and a preset number of special coordinate lexical units.

[0055] Steps 201 to 202 are described in detail below.

[0056] In some embodiments, to transform raw input data from different modalities into a feature representation that can be uniformly processed by a multimodal language model, after obtaining the image and visual task prompt text, the image needs to be input into the visual encoder of the multimodal language model for visual feature extraction to obtain image features, and the visual task prompt text needs to be input into the text encoder of the multimodal language model for visual task prompt text lexical extraction to obtain visual task prompt text lexical units. Specifically, the visual encoder is a neural network module specifically designed to process visual information. Its function is to transform the pixel information of the input image data into a high-dimensional vector that can represent the image content, i.e., image features. Similarly, the text encoder is a neural network module specifically designed to process language information. It transforms the visual task prompt text into visual task prompt text lexical units that can represent its semantics. Through this step, the raw, unstructured image and text data are transformed into structured feature vectors that can be fused and understood by downstream models.

[0057] Next, the image features and visual task prompt text terms are input into the trained multimodal language model. Utilizing the autoregressive mechanism of the multimodal language model, the predicted target text terms and their corresponding target coordinate terms in the image data are obtained. In this step, the multimodal language model fuses the obtained image features and visual task prompt text terms as unified contextual information for data autoregressive processing. This data autoregressive processing is a token-by-token autoregressive generation mechanism, where the model predicts the next most likely term based on the existing context and all generated terms. Specifically, in this embodiment, the vocabulary within the multimodal language model stores coordinate encoding mappings between each pixel coordinate in the image and a preset number of special coordinate terms. This is a mapping established between each pixel coordinate in the image's physical space and discrete coordinate terms in the model's vocabulary. It is based on this coordinate encoding mapping relationship that the model is able to predict not only the target text words representing the object category, but also the target coordinate words representing its precise spatial location when performing autoregressive word prediction, thus transforming a visual perception task into an end-to-end word sequence prediction task for a language model.

[0058] The process of generating coordinate encoding mapping relationships will be described further below.

[0059] The multimodal language model in this application also includes an image preprocessing module, as shown in the reference. Figure 3 The coordinate encoding mapping relationship is generated using the image preprocessing module, including the following steps 301 to 303.

[0060] Step 301: Normalize the coordinates of each pixel in the image according to the image size to obtain the corresponding normalized pixel coordinates.

[0061] Step 302: Based on a preset number of discrete integers, discretize each normalized pixel coordinate to obtain the corresponding coordinate quantization value. Each coordinate quantization value corresponds to an image coordinate term.

[0062] Step 303: Use the image preprocessing module to map the normalized pixel coordinates to image coordinate terms, and add the image coordinate terms to the vocabulary of the multimodal language model to generate coordinate encoding mapping relationships; wherein, a sequence of two image coordinate terms can be used to represent a normalized pixel coordinate.

[0063] Steps 301 to 303 are described in detail below.

[0064] In some embodiments, to generate a suitable coordinate encoding mapping, it is necessary to eliminate image size dependence and map the coordinates to a unified relative space. Therefore, it is first necessary to normalize the coordinates of each pixel in the image data according to the image size to obtain the corresponding normalized pixel coordinates. Specifically, the normalization process usually converts the pixel coordinates (x_pix, y_pix) on an image with a width of W and a height of H into a floating-point value in the interval [0, 1] using the formulas x_norm = x_pix / (W-1) and y_norm = y_pix / (H-1), i.e., the normalized pixel coordinates (x_norm, y_norm), providing a standardized input for subsequent discretization processing.

[0065] It is important to note that the denominators are W-1 and H-1 because pixel coordinates are indexed starting from 0. This ensures that the bottom right corner of the image (W-1, H-1) can be accurately mapped to normalized pixel coordinates of (1.0, 1.0).

[0066] Next, in order to discretize the continuous normalized coordinates so that they correspond to the finite lexical set of the multimodal language model, this embodiment discretizes each normalized pixel coordinate based on a preset number of discrete integers to obtain the corresponding coordinate quantization value. In this step, the preset number (e.g., 1000) of discrete integers defines the granularity of quantization. For example, the floating-point value in the normalized interval [0, 1] is divided into 1000 equal "grids" so that it can correspond to the finite lexical set. The discretization process is to multiply the normalized pixel coordinate (floating-point number) obtained in the previous step by the preset discrete integer (e.g., 999) and round it down to map it to an integer in the interval [0, 999]. This integer is the coordinate quantization value (x_q, y_q), that is, mapping the normalized coordinate (x_norm, y_norm) to the integer (x_q, y_q) in the interval [0, 999]. x_q = round(x_norm * 999), y_q = round(y_norm * 999). This step transforms continuous coordinate information into quantized coordinate values ​​that can be counted and indexed. This is equivalent to dividing each dimension of the image into a 1000-cell logical grid, with each quantized integer representing which "logical cell" the coordinate falls into. A two-dimensional image is thus divided into 1000 × 1000 logical cells.

[0067] Next, the image preprocessing module maps normalized pixel coordinates to image coordinate terms, and adds these terms to the vocabulary of the multimodal language model to generate coordinate encoding mappings. A sequence of two image coordinate terms can represent a normalized pixel coordinate, which is a key step in completing the coordinate encoding mapping and integrating it into the language model. A dedicated image preprocessing module is responsible for performing this coordinate encoding mapping task. First, this module assigns a unique, symbolic image coordinate term (e.g., ...) to each coordinate quantization value obtained in the preceding steps (e.g., discrete integers from 0 to 999).<coord_0> to<coord_999> This completes the mapping from each pixel coordinate of the image to image coordinate terms. Secondly, and crucially, this module formally adds these newly created image coordinate terms to the vocabulary of the multimodal language model. This is equivalent to expanding the model's "language," enabling it to understand and generate "new vocabulary" representing spatial locations. Finally, this step establishes the final representation method for accurately representing a two-dimensional normalized pixel coordinate using a sequence of two image coordinate terms, successfully transforming continuous pixel position information into a sequence of terms that can be directly processed by the language model.

[0068] The total number of coordinate lexical units (e.g., 1000) is determined by the quantization range defined by a preset discrete integer, ensuring that all possible coordinate quantization values ​​have a corresponding coordinate lexical representation, thus establishing a direct and stable one-to-one correspondence. In this application, the last 1000 lexical units are pre-adjusted or reserved as "special coordinate lexical units" in the vocabulary of the multimodal language model. These 1000 special coordinate lexical units establish a one-to-one correspondence with integers from 0 to 999. For example, the coordinate quantization value 120 corresponds to the special coordinate lexical unit.<coord_120> The coordinate quantization value of 512 corresponds to a special coordinate word.<coord_512> .

[0069] Through the above steps, a raw pixel coordinate (x_pix, y_pix) is successfully encoded into a sequence consisting of two special coordinate terms, for example...<coord_x_q><coord_y_q> .

[0070] Through steps 301 to 303 above, a mapping space connecting the continuous two-dimensional pixel coordinates of the image and the discrete one-dimensional word sequence of the language model is constructed. This allows the inherently complex visual localization task to be uniformly and elegantly transformed into the "next word prediction" problem, which is best suited for multimodal language models, without the need to design complex dedicated network structures. By transforming the highly difficult continuous coordinate regression problem into a relatively simple classification problem of selecting from a limited number of words, the learning and optimization difficulty of the model is greatly reduced. This representation method is extremely efficient; for example, only 4 coordinate words are needed to represent a bounding box, which significantly improves the processing speed. Ultimately, the entire visual task data processing method has both powerful language understanding capabilities and efficient and accurate localization capabilities.

[0071] Reference Figure 4 This is a schematic diagram illustrating the process of data autoregression processing using a multimodal language model, as provided in an embodiment of this application. Figure 4The diagram illustrates a specific implementation architecture for autoregressive data processing using a multimodal language model. The process begins by receiving two parallel input streams: firstly, the input image is fed into a visual encoder to extract multidimensional image features that characterize the image content; secondly, natural language task cues containing specific tasks (such as object detection, referential detection, point selection, or GUI clicks) are fed into a text encoder to extract corresponding visual task cue text terms. Subsequently, these two sets of features from different modalities are input together into the core multimodal language model (Qwen2.5-3B in this example) for fusion and understanding. Finally, the model, through the autoregressive mechanism of a large language model, generates a unified output terminology sequence word-by-word. This output terminology sequence contains key target coordinate terms, thus transforming a complex visual perception task into an end-to-end terminology sequence prediction problem.

[0072] In one example, a multimodal language model (MLLM), such as Qwen2.5-3B, is used to output a predicted sequence in an autoregressive manner. This sequence includes the output text and a sequence of special coordinate terms. For instance, in a visual task involving object detection, the output text represents the category. If the category is "person," and the input image contains two target objects belonging to the "person" category, the corresponding bounding box coordinates would be... <23> <123> <52> <77> , <123> <621> <999> <999> If the category is "seagull", then the input image includes two points belonging to the "seagull" category with coordinates of... <27> <61> , <69> <33> The corresponding output sequences are shown in the examples below.

[0073] "<|object_ref_start|>person<|object_ref_end|><|box_start|><coord_23><coord_123><coord_52><coord_77> ,<coord_123><coord_621><coord_999><coord_999> <|box_end|>, <|object_ref_start|>seagull<|object_ref_end|><|box_start|><coord_27><coord_61> ,<coord_69><coord_33> <|box_end|>.

[0074] The process of multimodal language models handling visual tasks is dynamic and flexible, and can naturally handle a variable number of targets. This application's autoregressive generative paradigm transforms the visual task into a language model's autoregressive "next lexical prediction" problem, thereby outputting category and bounding box information as a text sequence. This brings end-to-end simplicity, eliminating the need for post-processing such as NMS in principle (the model can learn to decide how many non-repeating targets to generate), and enabling seamless handling of various tasks such as object detection, point selection / keypointing in a unified manner—something traditional separate "head" architectures struggle to achieve.

[0075] Through steps 201 to 202 above, a clear and efficient multimodal information processing pipeline was constructed. By employing dedicated visual encoders and text encoders, high-quality and in-depth feature extraction of different modal information was ensured, laying a solid foundation for subsequent fusion processing. Subsequently, a powerful data autoregressive processing mechanism was used to transform the visual localization task into the language model generation framework, avoiding the complex dedicated detection heads or post-processing modules in traditional methods. Through the built-in coordinate encoding mapping relationship, the model can directly generate coordinate units representing spatial locations in a principled and interpretable manner, thereby effectively combining the model's powerful language understanding ability with accurate visual spatial localization ability, providing core technical support for the advancement and effectiveness of the entire visual task data processing method.

[0076] In some embodiments, target text terms represent the target text to be processed in the visual task, i.e., answering the question "what is it?". Correspondingly, target coordinate terms are the spatial portion of the output sequence, representing the location of the target object region in the image corresponding to the object described by the target text, i.e., answering the question "where is it?". This fundamental division of labor constructs a clear and structured output paradigm for this invention, decoupling and organically combining information from the two dimensions of semantic understanding and spatial localization within the same sequence.

[0077] In some embodiments, for the "general object detection" task, the target text lexical unit is typically a concise category phrase lexical unit, such as "person" or "seagull". However, for the "object reference object detection" task, which requires deeper semantic understanding, the target text lexical unit can be a complete natural language reference sentence lexical unit, such as "the man wearing the blue jacket". Similarly, for the "visual cue object detection" task, the target text lexical unit corresponds to a visual cue sentence lexical unit.

[0078] Furthermore, this framework is also applicable to "text recognition" tasks. In this scenario, the target text terminology is the specific text terminology recognized from the image (such as the text on a road sign), while the corresponding target coordinate terminology precisely defines the coordinates of the text region in the image. This design allows the model not only to "see" the location of the text but also to "understand" its content, seamlessly integrating object detection and optical character recognition (OCR) capabilities.

[0079] By functionally defining and classifying target text lexical units and target coordinate lexical units, this application can flexibly handle a variety of visual tasks, ranging from simple category detection to complex semantic reference and even character recognition, using a lexical sequence generation method, without changing the underlying architecture of the model or adding dedicated processing modules. This not only greatly simplifies the system design but also leverages the unique advantages of multimodal language models in deep semantic understanding, demonstrating their enormous potential in solving diverse visual perception problems.

[0080] The following section will further describe how the multimodal language model used in this application was trained.

[0081] Reference Figure 5 This is a schematic diagram of a training architecture based on a multimodal language model provided in an embodiment of this application. Figure 5 The diagram illustrates the systematic technical framework of a visual perception task processing method based on a multimodal language model disclosed in this application. This framework consists of three core modules working together. First, in the "task modeling" module, this application unifies the visual localization problem into a sequence generation task of "predicting specific coordinate lemmas" using a "multimodal language model." Second, to support model training, this application designs a powerful "data engine" module, which includes a "basic localization data engine" for generating basic labeled data and an "object reference data engine" for generating complex semantic data, automatically generating multimodal data samples. Finally, in the "training method" module, the model is trained using a two-stage process: first, it learns basic capabilities through large-scale "supervised fine-tuning," and then it optimizes and improves localization accuracy and reliability through "reinforcement learning post-training." The organic combination of these three modules constitutes the complete technical solution for training multimodal language models in this application.

[0082] Reference Figure 6 The training steps for a multimodal language model include steps 601 to 606.

[0083] Step 601: Obtain multiple multimodal data samples. Each multimodal data sample includes image sample data and corresponding label text information and label lexical coordinates.

[0084] Step 601 will be described in detail below.

[0085] In some embodiments, to enable a multimodal language model (MLLM) to learn an accurate mapping from natural language to image spatial locations, a large-scale, semantically rich, and accurately labeled training dataset is required. Existing publicly available datasets often fall short of this requirement in terms of scale or annotation quality. Therefore, the first step is to acquire multiple multimodal data samples, each including image sample data and corresponding label text information and label word coordinates. This is the data preparation stage of the entire model training process. This step aims to provide high-quality, large-scale training material for subsequent supervised fine-tuning and reinforcement learning. The image data samples and text data samples together constitute the model's input, while the label text information and label word coordinates together constitute the ground truth output that the model needs to learn. Label word coordinates refer to the encoded real pixel coordinates.

[0086] Furthermore, during the training of the multimodal language model, after acquiring the image sample data and corresponding label text information from the multimodal data samples, it is necessary to input the image sample data into the image preprocessing module to establish the coordinate encoding mapping relationship between pixel coordinates in the image and image coordinate words. Specifically, the image preprocessing module discretizes the normalized pixel coordinates in the image sample data and maps them into image coordinate word vectors. This allows for further transformation of the sample pixel coordinates corresponding to the label text information to obtain the corresponding label coordinate word sequence. The label coordinate words are then added to the vocabulary of the multimodal language model. A sequence of two label coordinate words can be used to represent a pixel coordinate.

[0087] Reference Figure 7 This is a schematic diagram illustrating the data autoregressive processing during the training of a multimodal language model, as provided in an embodiment of this application. Figure 7The diagram illustrates a specific implementation architecture for autoregressive data processing during the training of a multimodal language model. The process begins by receiving three parallel input streams: First, the input image sample data is fed into an image preprocessing model to establish a coordinate encoding mapping between pixel coordinates and image coordinate words in the image sample data. This mapping is then used to map the pixel coordinates in the image sample data into image coordinate word vectors to obtain the label coordinate word sequence corresponding to the label text. Second, the input image sample data is fed into a visual encoder to extract multidimensional image features that characterize the image content. Third, natural language task prompts containing specific tasks (such as object detection, referential detection, point selection, or GUI click) are fed into a text encoder to extract the corresponding visual task prompt text words. Subsequently, these two sets of features extracted from different modalities are input together into the core multimodal language model (Qwen2.5-3B in this example) for fusion and understanding. Ultimately, the model generates a unified training prediction lexical sequence one by one through an autoregressive mechanism. This facilitates the subsequent calculation of loss based on the pixel coordinates obtained after dequantization of the label coordinate lexical sequence and the training prediction lexical sequence, thereby enabling supervised fine-tuning training of the multimodal language model. Alternatively, it can calculate the reward value based on the pixel coordinates obtained after dequantization of the label coordinate lexical sequence and the training prediction lexical sequence, thereby enabling reinforcement learning of the supervised fine-tuned multimodal language model.

[0088] Understandably, before the training process of a multimodal language model begins, the image preprocessing module is first needed to generate label terminator coordinates as the "standard answer" for the training data. Specifically, the image preprocessing module first establishes a coordinate encoding mapping relationship between pixel coordinates in the image and special image coordinate terms. This process includes: normalizing each pixel coordinate according to the size of the image sample data; then, discretizing the normalized coordinates based on a preset number of discrete integers (e.g., 1000) to obtain the corresponding coordinate quantization values; subsequently, mapping these coordinate quantization values ​​to unique image coordinate terms, and formally adding these newly generated terms to the vocabulary of the multimodal language model. Through this series of steps, the original pixel coordinates associated with the label text information in the training samples are transformed into a sequence of label coordinate terms, each representing a pixel coordinate, that the multimodal language model can directly understand and learn. This sequence will serve as the ground truth in the subsequent supervised fine-tuning stage, used to calculate the loss and guide the model's learning.

[0089] This application designs an automated data engine to generate high-quality training samples of triples {image, text, special coordinate word sequence} on a large scale in a low-cost and scalable manner, as described below.

[0090] Reference Figure 8 When the visual task is object detection, the steps for acquiring multimodal data samples include the following steps 801 to 805.

[0091] Step 801: Obtain multiple public image data samples, and filter the multiple public image data samples based on preset resolution and preset compliance constraints to obtain compliant image data samples.

[0092] Step 802: Input each compliant image data sample into the initial multimodal language model to obtain the text description information corresponding to each compliant image data sample.

[0093] Step 803: Extract at least one basic category name from each text description information, and input each compliant image data sample and all corresponding basic category names into the target detection model for data detection to obtain the basic detection results of the basic category names in each compliant image data sample.

[0094] Step 804: Convert the basic detection results to obtain the corresponding tag word coordinates.

[0095] Step 805: Based on the compliant image data samples, basic category names, and tag lexical coordinates, obtain the multimodal data samples for the target detection task.

[0096] Steps 801 to 805 are described in detail below.

[0097] In some embodiments, when the visual task is object detection, the corresponding multimodal data samples are acquired and automatically generated using the grounding data engine for object detection. The grounding data engine for object detection in this application automatically generates "image + object category phrase + object detection bounding box" data. Output format: {image, [category phrase], [special coordinate lexical sequence]}.

[0098] Therefore, the initial step in data preparation is to first acquire a large number of public image data samples (such as those from COYO and SAM), and then filter these samples based on preset resolutions and compliance constraints to obtain compliant image data samples. This step aims to select original materials that meet quality requirements from massive public datasets (such as COYO and SAM). Specifically, preset resolutions are used to remove low-quality and blurry images, while preset compliance constraints are used to exclude inappropriate content. This initial cleaning and filtering process ensures that the subsequent automated annotation process only processes high-quality compliant image data samples, laying the foundation for generating high-quality training data.

[0099] Next, each compliant image data sample is input into the initial multimodal language model to obtain the corresponding text description information, which is the step in automatically generating the original text material. In this step, a powerful initial multimodal language model (e.g., Qwen2.5-VL-7B-Instruct) is used to generate rich and detailed natural language descriptions for each selected image. These descriptions (captions) typically include multiple objects in the image, the attributes of the objects (color, state, etc.), and the spatial relationships between them, providing rich text material for subsequent extraction of structured information. Understandably, a powerful initial multimodal language model (e.g., Qwen2.5-VL-7B-Instruct) is used as a tool for "image description." This model can generate rich and detailed natural language descriptions, i.e., text description information, for each input compliant image data sample, containing multiple objects, attributes, and their spatial relationships. This information provides rich corpus for subsequent extraction of structured target category names.

[0100] Next, to ensure the quality of the final annotation, which is also the core innovation that distinguishes this application from traditional data generation processes, we first use standard Natural Language Processing (NLP) toolkits (such as SpaCy) to automatically extract all noun phrases from the text description information generated in the previous step. For example, from the description "There are cut yellow lemons and green lemons on a table," we can extract "table," "cut yellow lemons," and "green lemons," and store the basic category names in the global phrase pool of the multimodal language model (the global phrase pool is composed of basic category names, such as cat, dog, etc.). Next, we introduce a crucial filtering step, the technical motivation of which is to address the inherent limitations of existing open-vocabulary detectors. Detectors like Grounding DINO or GLIP, while powerful, suffer from decreased accuracy and recall when understanding phrases modified by complex adjectives or attributes. For example, when the instruction is "locate green lemons," the model may incorrectly select all lemons (including yellow lemons) in the image. This problem of "inaccurate referencing" introduces a large amount of label noise into the training data, severely misleading the learning process of MLLM and preventing it from establishing an accurate correspondence between text and visual regions. To eliminate this ambiguity at its source, this application designs a set of filtering rules that retain only core noun phrases that represent the basic category of the object, while filtering out complex phrases with adjectives, adverbs, and other modifiers. In the example above, both "cut yellow lemon" and "green lemon" would be filtered out, and the system would refine and retain their common basic category name—"lemon." Similarly, the basic category name "table" would also be retained.

[0101] Therefore, in this process, extracting at least one basic category name from each text description and inputting each compliant image data sample and its corresponding basic category names into the object detection model for data detection, thus obtaining the basic detection results for the basic category names in each compliant image data sample, is the core step in generating basic localization data. A key technical feature of this step lies in the process of extracting basic category names: it not only uses natural language processing tools to extract noun phrases, but more importantly, it filters out complex phrases with modifiers such as adjectives and adverbs, retaining only the nouns of the basic categories of the target detection objects. This approach aims to eliminate label noise that may be introduced by complex descriptions, significantly improving data quality. Subsequently, these high-purity basic category names, along with the images, are input into the object detection model to obtain reliable basic detection results, namely, the triplet data of "image-basic category-bounding box".

[0102] Next, each basic detection result and its corresponding compliant image data sample are input into the initial multimodal language model to obtain the corresponding referential text information, which is the step in generating complex semantic descriptions. This step aims to upgrade the simple basic category names from the previous steps into descriptive statements that are more in line with human expression habits and contain rich context. By inputting images with bounding box annotations (i.e., basic detection results and compliant image data samples) into the initial multimodal language model and designing specific prompts, the model can be guided to generate natural and diverse referential text information such as "the puppy in the lower right corner of the image" or "the man wearing the blue jacket".

[0103] Therefore, in this process, the high-confidence basic category phrases selected in the previous step are input into an advanced open vocabulary object detector (such as DINO-X). The detector generates one or more corresponding bounding boxes for each phrase to obtain the basic detection results (i.e., the bounding box pixel coordinates of the target) corresponding to each basic category name in the compliant image data sample.

[0104] After obtaining the basic detection results, the results (i.e., the pixel coordinates of the bounding box of the target) are converted into label term coordinates that the model can directly process through a defined coordinate encoding process (including normalization, discretization, and mapping). Finally, the compliant image data samples, the corresponding basic category names, and the converted label term coordinates are combined to obtain a unified, multimodal data sample that can be directly used for model training and is designed for target detection tasks.

[0105] Through steps 801 to 805 above, an innovative cascaded arrangement of multiple AI models has enabled a fully automated and scalable target detection data production line. This addresses the industry pain points of high cost and long cycle of traditional manual annotation. By proactively performing noise reduction during the data generation stage—that is, extracting and using only the "basic category name" for target localization—the signal-to-noise ratio and annotation quality of the training data have been fundamentally improved. This ensures that the model can learn more accurate and robust visual-language correspondences, providing a massive amount of high-quality multimodal data samples for target detection tasks for subsequent model training. It is one of the key foundations and core assets for realizing high-precision visual perception models.

[0106] Reference Figure 9 When the visual task is a referential object task, the steps for acquiring multimodal sample data include the following steps 901 to 905.

[0107] Step 901: Obtain initial multimodal sample data and input the initial multimodal sample data into the initial multimodal language model to obtain the corresponding referential language text information. The initial multimodal sample data is image sample data including basic bounding boxes and category labels.

[0108] Step 902: Input the referential language text information and the corresponding image sample data into the preset target referential model for data processing to obtain the coordinate points of the target referential object.

[0109] Step 903: Generate object segmentation masks for all objects in the image sample data, and perform test matching between the coordinates of the target object and the bounding boxes corresponding to multiple object segmentation masks to obtain the test matching results.

[0110] Step 904: When the test matching result indicates that the coordinates of the target referent are in the target object segmentation mask, generate the association information between the coordinates of the target referent and the target object segmentation mask. The target object segmentation mask is one of multiple object segmentation masks.

[0111] Step 905: Based on the image sample data, the referential language text information, and the bounding boxes corresponding to the target object segmentation masks, obtain the multimodal data samples of the referential object task.

[0112] Steps 901 to 905 are described in detail below.

[0113] In some embodiments, when the visual task in this application is an object referencing task, after acquiring the corresponding multimodal data samples, the referencing data engine is used for automatic generation. After the model has mastered basic object detection capabilities, in order to enable it to understand more complex natural language instructions that are more in line with human expression habits (e.g., "the girl on the far left of the photo with her hand raised"), a dedicated "referring expression" dataset is needed for training. However, building such datasets has historically relied on costly and time-consuming manual annotation, resulting in existing datasets generally being small in scale and limited in content diversity. To overcome this bottleneck, this application proposes a fully automated object referencing data generation engine. This engine innovatively cascades and orchestrates the capabilities of multiple advanced AI models, generating high-quality {image, natural language referencing sentence, target bounding box} data on a large scale without any manual intervention, and outputting it in the format of {image, [natural language referencing sentence], [special coordinate lexical sequence]} triples. This automated process is unprecedented and greatly improves the production efficiency of training data for complex semantic understanding.

[0114] In this process, the process first acquires initial multimodal sample data, which is image sample data that already contains basic bounding boxes and category labels. Subsequently, this initial multimodal sample data is fed into a powerful initial multimodal language model. This model, through its strong image-based descriptive and language organization capabilities, generates referential text information for one or more targets in the image that conforms to human language habits and has rich contextual descriptions, such as "the man wearing the blue jacket."

[0115] In one example, for an image already labeled with basic target bounding boxes and corresponding category labels (which may come from public datasets such as O365, OpenImages, or generated by the aforementioned "basic localization data engine"), the image, the bounding boxes of all targets in the image, and their category labels are input into a powerful multimodal language model (e.g., Qwen2.5-VL-7B). This application incorporates specific prompts to guide the model in generating natural language reference sentences for one or more targets in the image, sentences that conform to human language habits and have rich contextual descriptions. The model outputs natural and diverse descriptive sentences such as "the man in the blue jacket" and "the puppy in the lower right corner of the image." This step leverages the powerful image-based description and language organization capabilities of LLM.

[0116] Furthermore, the system will perform a conversion from language description to spatial localization. The generated referential text information and corresponding image sample data will be input together into a pre-defined, specialized referencing model (e.g., Molmo) for data processing. The core capability of this referencing model is to "understand" a complex description and output its most crucial coordinate point on the image—the coordinate point of the target referent. This step successfully transforms the unstructured natural language description into a structured, single spatial pixel coordinate point on the image.

[0117] Next, reference objects are prepared for subsequent accurate matching, based on which object segmentation masks are generated for all objects in the compliant image data samples. This step utilizes a powerful Segment Anything Model (SAM) to process the input compliant image data samples. An object segmentation mask is pixel-level annotation information that accurately delineates the contours of objects. By generating high-precision segmentation masks for all original, known target objects in the image, this step provides accurate geometric boundary criteria for subsequent determination of coordinate point attribution.

[0118] Furthermore, the core decision-making step in the association process involves testing and matching the coordinates of the target reference object with the bounding boxes corresponding to multiple object segmentation masks to obtain the test matching results. This step performs a crucial "point within polygon" test. Specifically, the system checks whether the obtained coordinates of the target reference object fall within the region of any of the generated object segmentation masks, that is, whether the pointing point generated by the target reference model in the previous step falls within the bounding box corresponding to the segmentation mask region generated by a certain SAM. The test matching result is the conclusion of this test, which clarifies the spatial relationship between the coordinates and the precise object contours in the image.

[0119] When the test matching result indicates that the coordinates of the target referent object are within the bounding box corresponding to the target object segmentation mask, association information between the coordinates of the target referent object and the target object segmentation mask is generated. The target object segmentation mask is one of multiple object segmentation masks and is the step in establishing the association based on the successful matching result. This step means that once the test matching in the previous step is successful, the system has successfully established a strong association between the natural language referent sentence, its corresponding target referent object's coordinates, and the unique, high-confidence target object segmentation mask containing that point. In other words, if the coordinates of the target referent object fall within a certain mask, the system has successfully established a unique, high-confidence association between the natural language referent sentence and the target bounding box corresponding to that mask. The association information is a record of this binding relationship.

[0120] Finally, based on the compliant image data samples, the corresponding referential language text information, and the coordinates of the target referent, multimodal data samples are obtained, which is the final step in generating complete training samples. This step is performed after the previous steps have successfully established the association information. The system combines the original compliant image data samples, the referential language text information, and the precise location information corresponding to the successfully matched target object (usually the bounding box corresponding to the object's segmentation mask) to generate a complete, high-quality multimodal data sample. This sample ultimately exists in the form of {image, natural language referent, precise bounding box}, and can be directly used for model training.

[0121] Through the fully automated three-step process of "generation-pointing-matching" described above, this data engine can continuously generate massive amounts of high-quality referential training samples and output them in the format of {image, [natural language referential sentence], [special coordinate lexical sequence]} triples. It not only solves the pain points of high cost and small scale of traditional methods, but more importantly, the data it generates far surpasses manual annotation in the richness and diversity of language expression, providing a crucial data foundation for training a next-generation object detection model with truly powerful language understanding capabilities.

[0122] Through steps 901 to 905 above, the capabilities of the language generation model, the target referencing model, and the image segmentation model are innovatively linked and arranged, thereby solving the annotation problem of complex data such as object referencing in a fully automated manner. This greatly reduces data acquisition costs and improves production efficiency. For the generation of multimodal sample data for the object referencing task, high-precision segmentation masks are used for matching verification to ensure that the generated training data has extremely high annotation accuracy and effectively avoids label noise. This process can continuously create training samples that far exceed manually annotated ones in terms of the richness and diversity of language expression, providing a crucial data foundation for training a next-generation object detection model with truly powerful complex language understanding capabilities.

[0123] Furthermore, this application provides a method for automatically converting existing bounding box annotations into point annotations. This process involves three steps: first, generating a corresponding segmentation mask for each existing bounding box; second, calculating the minimum bounding rectangle of the mask; and finally, taking the intersection of the diagonals of this rectangle as the representative point. This automated process can efficiently generate large amounts of point-level annotation data to train models for advanced tasks requiring precise pointing.

[0124] In addition, this application also provides a scene text recognition data engine for generating image-text data with polygonal text regions and text transcription. The engine first uses an OCR algorithm (such as PaddleOCR) to detect text regions in an image and extract their polygonal boundaries. Then, while calculating the minimum bounding rectangle as the bounding box, it records the recognized text content. This engine can provide the model with a large number of training samples for scene text recognition, enabling it to learn the ability to locate and recognize text in complex visual environments.

[0125] Step 602: Input the multimodal data samples into the initial multimodal language model and use the autoregressive mechanism of the initial multimodal language model to generate the first training prediction word sequence.

[0126] Step 602 will be described in detail below.

[0127] Based on the obtained multimodal data samples, the initial multimodal language model is further trained. In this application, the training process adopts a two-stage strategy: Stage 1: Supervised Fine-Tuning (SFT) – enabling the model to acquire basic coordinate prediction capabilities; Stage 2: GRPO-based Reinforcement Post-Training – addressing the geometric mismatch and repetitive prediction problems caused by discretized coordinates, improving detection accuracy and output quality. This "teacher-guided, then self-optimized" design ensures that the model possesses both language understanding and accurate localization capabilities under open vocabulary conditions.

[0128] In the first phase of training, the training data consisted of approximately 22 million images and their corresponding coordinates and annotations generated by the aforementioned data engines (Grounding, Referring, Pointing, OCR, etc.) and public datasets, resulting in a large number of multimodal data samples. Each training data set was in the form of: input data <text, image> and corresponding label data <annotation information + special coordinate word sequence>.

[0129] The first stage of the training process then begins with forward inference, whereby the multimodal data samples are input into the initial multimodal language model, and the autoregressive mechanism of the initial multimodal language model is used to generate the first training prediction token sequence. The autoregressive mechanism is a token-by-token generation method, whereby the model predicts the next most likely token based on the input image and text context, as well as all the tokens that have already been generated.

[0130] Reference Figure 10The multimodal data samples are input into an initial multimodal language model, and the first training prediction word sequence is generated using the autoregressive mechanism of the initial multimodal language model, including the following steps 1001 to 1003.

[0131] Step 1001: Obtain category names from multiple multimodal data samples and obtain a global phrase pool based on all category names.

[0132] Step 1002: For each multimodal data sample, select at least one category name from the global phrase pool that does not exist in the image sample data as a negative query sample.

[0133] Step 1003: Input the multimodal data samples and negative query samples into the initial multimodal language model, and use the autoregressive mechanism of the initial multimodal language model to generate the first training prediction word sequence; the first training prediction word sequence corresponding to the negative query sample is a specific sequence of "coordinates not found".

[0134] Steps 1001 to 1003 are described in detail below.

[0135] First, task templates are constructed: various question-and-answer templates are designed, such as "Detect [PHRASE] in thisimage", to simulate real user needs.

[0136] Then, considering that user commands often involve multiple targets in real-world scenarios, during training, if an image contains N labeled object detection categories, with the label data being <labeled category, special coordinate term sequence>, we randomly sample 1 to N object categories, combine them into a single, longer text command, and require the model to output the coordinate sequence of all corresponding object detection objects in sequence. This trains the model's ability to handle complex, multi-target commands.

[0137] Therefore, to create a comprehensive collection containing all object phrases that appear in the training dataset (i.e., the global phrase pool for the multimodal language model), the first step is to obtain category names from multiple multimodal data samples and derive the global phrase pool based on all basic category names. This is a preparatory step for implementing the negative sample training strategy. In this process, by summarizing the category names from all multimodal data samples, the phrase pool becomes a queryable "complete set of object categories," providing a data foundation for accurately extracting object category names not included in any specific image sample in subsequent steps.

[0138] Furthermore, for each multimodal data sample, at least one category name that does not exist in the image sample data is selected from the global phrase pool as a negative query sample. The purpose of this step is to create a training scenario for the model that requires it to detect objects that do not exist, thereby teaching the model the ability to "refuse to recognize". Specifically, the system randomly selects one or more (e.g., 1 to 10) category names that do not belong to the current multimodal data sample (i.e., image) from the global phrase pool established in the previous step, and constructs them into query phrases, which are the "negative sample query instructions".

[0139] After preparing the positive and negative samples, the multimodal data samples and negative query samples are input into the initial multimodal language model. The autoregressive mechanism of the multimodal language model is then used to generate the first training prediction word sequence. In this step, the initial multimodal language model simultaneously receives the multimodal data samples containing information about real objects, as well as negative query samples containing queries about objects that do not exist in the graph. The first training prediction word sequence corresponding to the negative query sample is a specific sequence of "coordinates not found". Through its autoregressive mechanism, the model generates a predicted output sequence for this complex input that mixes positive and negative queries. This sequence is the first training prediction word sequence, which reflects the model's comprehensive judgment ability on the existence and non-existence of objects in the current training state.

[0140] Understandably, multimodal language models (MLLMs), as generative models, inherently "always try to provide an answer," which can easily lead to "hallucination" in visual perception tasks—that is, the model "creates" objects that do not exist in the image and generates a seemingly reasonable bounding box for them. This behavior is fatal for a detection system that requires factual accuracy. To fundamentally suppress this illusion, this application designs a negative sample training strategy, which essentially teaches the model to explicitly "refuse" answers. First, all object phrases that have appeared in all 22 million training samples are collected to construct a large "global phrase pool." When processing each training image, 1 to 10 target objects that are determined not to exist in the current image are randomly selected from this global phrase pool, and negative sample query instructions are generated. These negative sample query instructions (e.g., in a picture of a cat, the query text is "elephant" category) do not require the model to not output anything, but rather train it to generate a structured, specific sequence that explicitly indicates "coordinates not found." Specifically, when the query object does not exist, the model needs to learn to output a sequence like <|object_ref_start|>negative sample query instruction<|object_ref_end|><|box_start|>None<|box_end|>. In this way, the model is no longer a system that only gives "affirmative" results. It learns to distinguish between the two distinct states of "target found" and "coordinates not found." When the model outputs a special coordinate term sequence of None, this is not a failed prediction, but a successful and correct judgment. This enhanced "refusal capability" is key to the high reliability and low false alarm rate achieved in this application, making it far more robust and reliable in practical applications than other models prone to illusions.

[0141] Through steps 1001 to 1003 above, the fatal flaw inherent in multimodal language models as generative models—their susceptibility to "illusions"—is resolved. By explicitly teaching the model how to identify and clearly "reject" non-existent object queries, the model is endowed with the ability to distinguish between the two distinct states of "target found" and "coordinates not found," greatly improving the model's reliability and robustness in open-set object detection scenarios and effectively suppressing the generation of false affirmations. This strategy makes the model no longer a system that can only "affirm," but a more realistic and credible visual perception tool.

[0142] Step 603: Based on the first training predicted word sequence and the labeled text information and labeled word coordinates, calculate the cross-entropy loss function, and perform supervised fine-tuning training on the initial multimodal language model to obtain the fine-tuned multimodal language model.

[0143] Step 603 will be described in detail below.

[0144] In some embodiments, the system further performs a first-stage parameter optimization, namely, calculating the cross-entropy loss function based on the first training predicted word sequence and label text information, and label word coordinates, to perform supervised fine-tuning training on the initial multimodal language model, resulting in a fine-tuned multimodal language model. The core of this step lies in using the cross-entropy loss function, which treats the coordinate prediction task as a classification problem, guiding the model's learning by calculating the difference between the predicted word sequence and the actual label word sequence. Through repeated supervised fine-tuning training, the final fine-tuned multimodal language model has mastered the basic ability of visual localization.

[0145] In the final step of training, the model predicts a discrete sequence of "specific coordinate terms" representing the coordinates. This application employs the standard cross-entropy loss function for supervised training. Essentially, this transforms the pixel coordinate prediction problem into a classification problem with 1000 coordinate terms. The model needs to select the most correct coordinate term for each pixel region from these 1000 possible terms. Through training on massive amounts of data, the model learns to accurately map specific pixel regions in an image to specific coordinate terms.

[0146] After the first stage of SFT training, the model learned the basic abilities of discrete "special coordinate word sequence prediction" and "open set category understanding". However, due to the geometric error brought about by discretization and the teacher's forced guidance on the output length, it is easy to generate repeated target detection boxes and coordinate offset problems. Therefore, this application adds GRPO reinforcement learning post-training.

[0147] Step 604: Obtain at least one sampled training data from multiple multimodal data samples, input the sampled training data into the fine-tuned multimodal language model, and use the autoregressive mechanism of the fine-tuned multimodal language model to generate a second training prediction word sequence.

[0148] Step 604 will be described in detail below.

[0149] Next, we will conduct the second stage of GRPO reinforcement learning training to solve the two core problems in the SFT stage: geometric bias (the discrete token classification is not completely consistent with the distance to the real pixels) and behavioral defects (repeated generation of the same box or over-prediction during inference).

[0150] Understandably, GRPO (Group Relative Policy Optimization) is an advanced reinforcement learning policy optimization algorithm primarily used for fine-tuning large generative models. Its core idea is that, given the same input instruction, the current model first autonomously generates a group containing multiple distinct complete output sequences (i.e., "rollouts"). Then, the algorithm uses a pre-defined reward function to evaluate the quality of each output sequence in this group and compares their quality, thus obtaining the relative advantage between the outputs. Finally, the algorithm uses policy gradient optimization to update the model's parameters based on the calculated relative advantage, increasing the probability of the model generating "better" sequences in the future. To ensure the stability of the training process, GRPO typically introduces KL divergence as a constraint to prevent the model from deviating too far from its basic capabilities learned during supervised fine-tuning, thereby achieving fine-grained and robust tuning of the model's behavior.

[0151] In this application, GPRO is used to enable a fine-tuned multimodal language model to autonomously generate multiple complete detection results (rollouts) given an image and text instructions.

[0152] Based on this, in the second stage, at least one sampled training data point is first obtained from multiple multimodal data samples. This sampled training data is then input into the fine-tuned multimodal language model, and the autoregressive mechanism of the fine-tuned multimodal language model is used to generate a second set of training prediction word sequences. This step utilizes the fine-tuned multimodal language model trained in the previous stage to autonomously generate multiple complete prediction results without relying on forced guidance. The generated second set of training prediction word sequences represents the model's autonomous performance on a given task under the current policy, providing a basis for subsequent reward evaluation.

[0153] Specifically, in this step, 12K training data (i.e., multimodal data samples) are randomly sampled from the SFT model. Then, each training data (i.e., multimodal data sample) is input into the fine-tuned multimodal language model for data autoregression processing. The fine-tuned multimodal language model will autonomously output a set of second training prediction word sequences (including <prediction information + prediction coordinate word sequences>).

[0154] Step 605: Based on the second training prediction word sequence and label text information, and label word coordinates, calculate a set of corresponding training reward values.

[0155] Step 605 will be described in detail below.

[0156] In some embodiments, after the model generates prediction results, the system performs a second-stage evaluation process. This involves calculating a set of training reward values ​​based on a set of second-trained predicted word sequences and the corresponding label text information and label word coordinates for each second-trained predicted word sequence. The system then compares the second-trained predicted word sequence with the actual label word coordinates. However, this comparison does not use cross-entropy loss; instead, it calculates a quantified score using one or more geometric perception reward functions (i.e., calculating the reward value based on the prediction result <prediction information + special coordinate word sequence> and the label data of the training data <annotation information + special coordinate word sequence>). This score is the training reward value, which directly measures the accuracy of the prediction result in geometric positioning.

[0157] In a preferred embodiment, a set of second training prediction word sequences is generated for each sampled training data in the fine-tuned multimodal language model (the number of words in a set is determined by the model hyperparameters, such as 4 or 8). Each second training prediction word sequence corresponds to a training reward value, and a set of reward values ​​is used to calculate the relative advantage value within the set.

[0158] The following section will further describe how to generate training reward values.

[0159] Reference Figure 11 Based on the second training prediction word sequence, the label text information, and the label word coordinates, a set of corresponding training reward values ​​is calculated, including the following steps 1101 to 1104.

[0160] Step 1101: When the second training prediction word sequence is the corresponding target detection box prediction task information, the predicted target detection box is calculated based on the second training prediction word sequence.

[0161] Step 1102: Based on the intersection-union ratio between the predicted target detection box and the label bounding box corresponding to the label lexical coordinates, obtain a set of training reward values ​​corresponding to the second training predicted lexical sequence.

[0162] Step 1103: When the second training prediction word sequence is the corresponding target point prediction task information, and the second training prediction word sequence points to an irregular target object, a sample object segmentation mask is generated based on the irregular target object in the corresponding image data sample, and a set of training reward values ​​corresponding to the second training prediction word sequence is obtained based on the matching result between the predicted target point in the second training prediction word sequence and the bounding box corresponding to the sample object segmentation mask.

[0163] Step 1104: When the second training prediction word sequence is the corresponding target point prediction task information, and the second training prediction word sequence points to a regular target object, obtain the object bounding box of the regular target object in the corresponding image data sample, and obtain a set of training reward values ​​corresponding to the second training prediction word sequence based on the matching result of the predicted target point and the object bounding box of the regular target object.

[0164] Steps 1101 to 1104 are described in detail below.

[0165] In some embodiments, when the second training prediction word sequence corresponds to the object detection box prediction task information, the predicted object detection box is calculated based on the second training prediction word sequence. This step first determines whether the current task is a task that requires outputting bounding boxes (e.g., object detection). If so, the system parses and decodes four coordinate words from the second training prediction word sequence (a sequence containing special coordinate words) generated by the model, and converts them into specific pixel coordinates on the image, thereby forming a complete predicted object detection box that can be used for geometric evaluation.

[0166] Then, based on the Intersection over Union (IoU) ratio between the predicted target detection boxes and the label bounding boxes corresponding to the label lexical coordinates, a set of training reward values ​​corresponding to the second training predicted lexical sequence is obtained. The label bounding boxes are obtained based on the label lexical coordinates, which is the core reward calculation step for the bounding box task. In this step, the IoU is a standard geometric metric used to measure the degree of overlap between two bounding boxes. The system calculates the IoU value between the predicted target detection boxes obtained in the previous step and the label bounding boxes decoded based on the real label lexical coordinates. This IoU result is directly used as a set of training reward values ​​corresponding to the second training predicted lexical sequence, thereby directly aligning the model's optimization objective with the final geometric localization accuracy.

[0167] Understandably, the Box IoU Reward (i.e., a set of training reward values ​​corresponding to the second training predicted lexical sequence) is a reward function specifically used to evaluate the localization accuracy of tasks that output bounding boxes. It is applicable to all tasks that predict detection boxes, including object detection and object referring. The core of this reward is calculating the Intersection over Union (IoU) ratio between the model-predicted bounding box and the ground truth bounding box.

[0168] When the second training prediction lexical sequence corresponds to the target point prediction task information, and the second training prediction lexical sequence points to an irregular target object, a sample object segmentation mask is generated based on the irregular target object in the corresponding image data sample. Based on the matching result between the predicted target point in the second training prediction lexical sequence and the bounding box corresponding to the sample object segmentation mask, a set of training reward values ​​corresponding to the second training prediction lexical sequence is obtained. This is the precise reward calculation step for the irregular object selection task. This step first uses the segmentation model to generate a pixel-level, precise sample object segmentation mask for the irregularly shaped target object. Then, by performing a "point within polygon" test, it is determined whether the predicted target point output by the model falls within the bounding box corresponding to the mask region. The matching result of this test (yes or no) is used to generate a set of training reward values ​​corresponding to the second training prediction lexical sequence; for example, a positive reward is given if the point is within the mask, and a zero or negative reward is given otherwise.

[0169] Understandably, the Point-in-Mask Reward (i.e., a set of training reward values ​​corresponding to the second training predicted word sequence) is awarded when a point is within the mask. This reward function is used to evaluate point selection tasks where the output is a coordinate point, especially when the target object has an irregular shape. It aims for the highest pointing accuracy. First, a pixel-level segmentation mask is generated using a segmentation model (such as SAM) as the baseline target. This mask accurately outlines the object's contour. Then, it is determined whether the model-predicted point coordinates (x, y) fall within the region of this mask. If the point is within the mask, a higher positive reward (e.g., +1) is given; if the point is outside the mask, a zero or negative reward (e.g., 0 or -1) is given.

[0170] When the second training prediction word sequence corresponds to the target point prediction task information, and the second training prediction word sequence points to a regular target object, the bounding box of the regular target object in the corresponding image data sample is obtained. Based on the matching result between the predicted target point and the bounding box of the regular target object, a set of training reward values ​​corresponding to the second training prediction word sequence is obtained. This is an efficient reward calculation step for regular object selection tasks. This step is a simplified but efficient version of the previous step, mainly applicable to scenarios where the target shape is regular (close to a rectangle) or the interaction area is clear (such as graphical user interface positioning). It directly uses the bounding box of the regular target object as the judgment region, determines the matching result by checking whether the predicted target point falls within the bounding box, and generates a set of training reward values ​​corresponding to the second training prediction word sequence accordingly.

[0171] Understandably, the Point-in-Box Reward (i.e., a set of training reward values ​​corresponding to the second training predicted word sequence) is a simplified but efficient version of the "point within the mask" reward, also used to evaluate tasks where the output is a single coordinate point. It is mainly suitable for scenarios where the target shape is regular (close to a rectangle) or the interaction area is well-defined, i.e., graphical user interface grounding.

[0172] Since a training sample typically corresponds to only one specific visual task (such as bounding box detection or irregular point selection), only one type of reward value will be calculated in the preceding steps. The purpose of this step is to select the corresponding set of training reward values ​​from all possible calculated reward values ​​based on the task type of the current sample, and use it as the final training reward value that will be used for subsequent model parameter optimization.

[0173] When training for visual tasks such as general object detection or text recognition, the system obtains training reward values ​​based on a set of training reward values ​​corresponding to the second training predicted word sequence. This choice is based on the fact that the core objective of these two types of tasks is to accurately define the extent of a two-dimensional region. Therefore, using a set of training reward values ​​corresponding to the second training predicted word sequence, calculated based on the intersection-over-union (IoU) ratio between the predicted object detection box and the label bounding box, is the most direct and effective metric for evaluating the model's performance on these tasks, and it can most effectively guide the model to learn to generate bounding boxes with accurate shape and size.

[0174] When training for visual tasks such as object referencing or visual cue detection, the system obtains a training reward value based on a set of training reward values ​​corresponding to the second training predicted word sequence. The logic behind this choice is that these tasks are essentially closer to precise "pointing" based on complex linguistic instructions. Therefore, using a reward value based on whether the predicted target point falls within the target region (segmentation mask or bounding box) allows for a more accurate evaluation of the model's "pointing" accuracy, thereby more effectively optimizing the model's spatial pointing ability for complex linguistic descriptions.

[0175] Through steps 1101 to 1105 above, the problem of "optimization target mismatch" caused by the use of cross-entropy loss in the prior art is solved. By designing a reward function directly related to the final task indicator (such as positioning accuracy), such as a set of training reward values ​​corresponding to the second training prediction word sequence based on the intersection-union ratio and a set of training reward values ​​corresponding to the second training prediction word sequence based on the matching result of points in the mask / a set of training reward values ​​corresponding to the second training prediction word sequence, the training process of the model is guided to directly optimize its geometric positioning accuracy. This can effectively bridge the gap between discrete classification and continuous regression, significantly reduce the geometric error caused by coordinate quantization, and thus significantly improve the prediction accuracy and reliability of the model in various visual perception tasks.

[0176] Step 606: Based on a set of training reward values ​​corresponding to the training visual task, perform reinforcement learning training on the fine-tuned multimodal language model to obtain the multimodal language model.

[0177] Step 606 will be described in detail below.

[0178] After obtaining the training reward value, the fine-tuned multimodal language model is trained using reinforcement learning based on at least one training reward value corresponding to the training visual task. This process, which involves policy optimization and model update, is crucial for obtaining the multimodal language model. This step utilizes the training reward value obtained in the previous step as the core feedback signal, and uses reinforcement learning algorithms such as policy gradients to optimize the parameters of the fine-tuned multimodal language model. Essentially, this process "rewards" policies that generate high-precision, correct predictions while "penalizing" policies that lead to low accuracy or repetitive predictions. Through this training phase, the basic capabilities learned in the SFT stage are significantly refined and calibrated, resulting in a multimodal language model that combines basic localization capabilities with high accuracy and reliability, as described below.

[0179] Reference Figure 12 The fine-tuned multimodal language model is trained by reinforcement learning based on a set of training reward values ​​corresponding to the training visual task to obtain the multimodal language model, including the following steps 1201 to 1202.

[0180] Step 1201: Standardize a set of training reward values ​​to obtain the relative advantage value within the group.

[0181] Step 1202: Based on the relative advantage value, use the policy gradient method with KL divergence constraint to train the fine-tuned multimodal language model through reinforcement learning to obtain the multimodal language model.

[0182] Steps 1201 to 1202 are described in detail below.

[0183] In some embodiments, to improve the stability of the training process, the training reward values ​​need to be standardized to obtain relative advantage values, which is a data preprocessing step before updating the reinforcement learning model. Since directly using the raw training reward values ​​for gradient updates may lead to training instability due to drastic fluctuations in the values, standardization transforms the original set of training reward values ​​corresponding to the visual task into relative advantage values ​​within that set. The relative advantage value not only reflects the quality of the current prediction result but also its superiority relative to the average performance, thus providing a more robust and less variance-laden guiding signal for subsequent policy gradient updates.

[0184] Then, based on the relative advantage value, the policy gradient method with KL divergence constraints is used to train the fine-tuned multimodal language model through reinforcement learning. Based on the fine-tuned multimodal language model after multiple training iterations, the final multimodal language model is obtained, which is the core optimization step in the reinforcement learning phase. In this step, the policy gradient method is a reinforcement learning algorithm that, guided by the relative advantage value obtained in the previous step, directly adjusts the parameters (i.e., the policy) of the fine-tuned multimodal language model to increase the probability of obtaining high-reward behaviors. The KL divergence constraint is a key technical feature that continuously measures the difference between the current model policy and the model policy after the supervised fine-tuning (SFT) phase during the optimization process, limiting this difference to a minimum. After multiple iterations of this training process, the final multimodal language model maintains the basic capabilities of the SFT phase while significantly improving geometric localization accuracy.

[0185] Through steps 1201 to 1202 above, the policy gradient method with KL divergence constraints solves the common technical problem of training instability when fine-tuning large language models. By standardizing the reward value to a relative advantage value, a smoother gradient signal is provided for model updates. Secondly, the core KL divergence constraint mechanism acts as a "stabilizing anchor," effectively preventing policy collapse or catastrophic forgetting in the process of pursuing higher rewards. This ensures that the model does not deviate from the solid foundation learned in the SFT stage while optimizing accuracy. This fine-tuning strategy, which balances exploration optimization and training stability, is the key to successfully applying reinforcement learning to improve the visual localization capabilities of multimodal large models. Ultimately, the trained model achieves ideal results in both accuracy and inference stability.

[0186] Through steps 601 to 606 above, the two-stage training process consisting of these multiple steps solves the fundamental technical problem of "optimization target mismatch" that occurs when multimodal language models perform visual tasks and are transformed into discrete classification tasks in existing technologies. The first stage of supervised fine-tuning training builds a solid classification foundation for the model, while the second stage of reinforcement learning training successfully bridges the gap between discrete classification and continuous regression by introducing geometric perception rewards that are directly related to the final task indicators. This "teacher-guided first, then self-optimized" design enables the final model to not only understand complex language instructions, but also achieve high-precision visual localization while suppressing undesirable behaviors such as repeated predictions and coordinate shifts, thus significantly improving the overall performance and reliability of the model.

[0187] Step 102: Decode all target coordinate words to obtain the target pixel coordinates in the image corresponding to the visual task prompt text.

[0188] Step 102 is described in detail below.

[0189] In some embodiments, after the model generates a complete output sequence, the final step is to decode the "special coordinate terms" in the sequence back to their physical pixel locations on the image. Therefore, decoding all target coordinate terms to obtain the target pixel coordinates corresponding to the visual task prompt text in the image is a crucial step in transforming the abstract information output by the model into concrete physical locations. The decoding process is the precise inverse operation of the coordinate encoding process. It first converts each target coordinate term back to its corresponding coordinate quantization value, and then, according to preset dequantization and denormalization formulas, calculates and restores this integer value to the absolute pixel coordinates (i.e., target pixel coordinates) that can be located on the original image data. This step remaps the discretized and symbolic location information output by the language model back to the continuous two-dimensional physical space of the image, providing the foundation for the final visualization result.

[0190] The following section will describe in more detail how to decode the target coordinate tokens.

[0191] Reference Figure 13 The process involves decoding all target coordinate units to obtain the target pixel coordinates in the image corresponding to the visual task prompt text, including the following steps 1301 to 1302.

[0192] Step 1301: Based on the vocabulary, convert the target coordinate lexicon into the corresponding coordinate quantization value.

[0193] Step 1302: Based on the coordinate encoding mapping relationship, restore the coordinate quantization value to the target pixel coordinates corresponding to the target coordinate words. The target pixel coordinates represent the target object region corresponding to the visual task prompt text.

[0194] Steps 1301 to 1302 are described in detail below.

[0195] In some embodiments, the target coordinate lexical units are first converted into corresponding coordinate quantized values ​​based on the vocabulary, where the vocabulary is a complete symbol-numerical mapping system stored internally by the multimodal language model. When the model generates, for example...<coord_23> When the target coordinate lexical is encountered, the decoding process queries the vocabulary and, based on the preset, unique correspondence within it, accurately converts the symbolized lexical back to its represented coordinate quantization value, i.e., 23. This step restores the linguistic symbols output by the model to a standardized integer format suitable for mathematical operations.

[0196] Then, based on the coordinate encoding mapping relationship, the quantized coordinate values ​​are restored to the target pixel coordinates corresponding to the target coordinate words. These target pixel coordinates represent the target object region corresponding to the visual task prompt text and are the final step in restoring discrete values ​​to physical coordinates in the image. The coordinate encoding mapping relationship here refers to the precise inverse operation formula of normalization and discretization processing during coordinate encoding. The system uses this coordinate encoding mapping relationship to perform inverse quantization and inverse normalization calculations on each obtained coordinate quantized value, combined with the size information of the original image, thereby accurately restoring it to the target pixel coordinates that can be located on the original image. Finally, the set of these target pixel coordinates accurately represents the target object region corresponding to the visual task prompt text, such as the four vertices of a bounding box.

[0197] In one example, the system will extract all target coordinate terms from the sequence, for example...<coord_23> ,<coord_123> ,<coord_52> ,<coord_77> Then, using the coordinate encoding mapping relationship, inverse quantization and inverse normalization calculations are performed to convert the quantized coordinate values ​​represented by each coordinate term (e.g., 23, 123, 52, 77) back to absolute pixel coordinates (x_pix, y_pix) corresponding to the original image size. These final pixel coordinates can then be used to display object detection boxes, mark key points / points, or outline polygons on the image, thus completing the visual perception task.

[0198] Through steps 1301 and 1302 above, the abstracted and symbolic location information output by the multimodal language model is accurately restored to physical coordinates usable in the visual world, providing the necessary technical path. This two-step decoding process, "symbol to integer, integer to coordinate," together with the innovative coordinate encoding mapping relationship of this application, constitutes a complete technical closed loop. It ensures that the model's powerful semantic understanding ability can be losslessly transformed into high-precision spatial positioning ability. This decoding process is a key link in realizing the entire end-to-end visual task processing method, making the model's output no longer just descriptive text, but concrete and operable spatial positioning information that can be directly used to draw bounding boxes or mark key points, thus providing a solid guarantee for the practicality and advancement of the entire technical solution.

[0199] Step 103: Based on all target pixel coordinates, obtain the target output result corresponding to the visual task prompt text.

[0200] Step 103 will be described in detail below.

[0201] Finally, based on the target text words and all target pixel coordinates, the target output corresponding to the visual task prompt text is obtained, integrating all the information obtained in the previous steps: semantic information provided by the target text words and spatial positioning information provided by the target pixel coordinates. Based on this integrated information, the system ultimately generates a target output that can complete the visual task and is intuitively perceived by the user. For example, it accurately draws the bounding box of the target on the original image data, marks key points, or delineates polygonal regions, thus successfully completing the end-to-end processing flow from natural language instructions to visual operations.

[0202] This application also provides a visual task processing device that can implement the above-described visual task processing method, see reference. Figure 14 The device 1400 includes: The prediction module 1410 is used to input images and visual task prompt text into a multimodal language model for visual task processing, and obtain an output word sequence consisting of target text words and target coordinate words; The pixel coordinate transformation module 1420 is used to decode all target coordinate words to obtain the target pixel coordinates in the image that correspond to the visual task prompt text. The result output module 1430 is used to obtain the target output result corresponding to the visual task prompt text based on the target pixel coordinates.

[0203] In some embodiments, the prediction module 1410 is further configured to: The image is input into the visual encoder for visual feature extraction to obtain image features, and the visual task prompt text is input into the text encoder for visual task prompt text word extraction to obtain visual task prompt text words. Image features and visual task-cue text lexical units are input into a multimodal language model, and the target text lexical units and output lexical sequences are generated using the autoregressive mechanism of the multimodal language model. The vocabulary of the multimodal language model stores the coordinate encoding mapping relationship between each pixel coordinate in the image and a preset number of special coordinate words.

[0204] In some embodiments, the prediction module 1410 is further configured to: Based on the image size, the coordinates of each pixel in the image are normalized to obtain the corresponding normalized pixel coordinates; Based on a preset number of discrete integers, each normalized pixel coordinate is discretized to obtain the corresponding coordinate quantization value. Each coordinate quantization value corresponds to an image coordinate word. The image preprocessing module is used to map normalized pixel coordinates into image coordinate terms, and the image coordinate terms are added to the vocabulary of the multimodal language model to generate coordinate encoding mapping relationships. In this context, a sequence of two image coordinate terms can be used to represent a normalized pixel coordinate.

[0205] In some embodiments, the pixel coordinate transformation module 1420 is further configured to: Based on the vocabulary, the target coordinate lexicon is converted into the corresponding coordinate quantization value; Based on the coordinate encoding mapping relationship, the coordinate quantization value is restored to the target pixel coordinates corresponding to the target coordinate words. The target pixel coordinates represent the target object region corresponding to the visual task prompt text.

[0206] In some embodiments, the prediction module 1410 is further configured to: Acquire multiple multimodal data samples, each of which includes image sample data and corresponding label text information and label lexical coordinates; Multimodal data samples are input into the initial multimodal language model, and the first training prediction word sequence is generated using the autoregressive mechanism of the initial multimodal language model. Based on the first training predicted word sequence and label text information, and the label word coordinates, the cross-entropy loss function is calculated, and the initial multimodal language model is subjected to supervised fine-tuning training to obtain the fine-tuned multimodal language model; At least one sampled training data is obtained from multiple multimodal data samples. The sampled training data is input into the fine-tuned multimodal language model. The autoregressive mechanism of the fine-tuned multimodal language model is used to generate a second training prediction word sequence. Based on the second training prediction word sequence and label text information, and label word coordinates, calculate a corresponding set of training reward values; The fine-tuned multimodal language model is trained using a set of training reward values ​​corresponding to the training visual task, and a multimodal language model is obtained.

[0207] In some embodiments, the prediction module 1410 is further configured to: Multiple public image data samples are acquired, and filtered based on preset resolution and preset compliance constraints to obtain compliant image data samples. Each compliant image data sample is input into the initial multimodal language model to obtain the text description information corresponding to each compliant image data sample; Extract at least one basic category name from each text description information, and input each compliant image data sample and all corresponding basic category names into the target detection model for data detection to obtain the basic detection results of the basic category names in each compliant image data sample; The basic detection results are converted to obtain the corresponding tag word coordinates; Based on compliant image data samples, basic category names, and tag term coordinates, multimodal data samples for the target detection task are obtained.

[0208] In some embodiments, the prediction module 1410 is further configured to: Acquire initial multimodal sample data and input the initial multimodal sample data into the initial multimodal language model to obtain the corresponding referential language text information. The initial multimodal sample data is image sample data including basic bounding boxes and category labels. The textual information of the referential language and the corresponding image sample data are input into the preset target referential model for data processing to obtain the coordinate points of the target referential object; Generate object segmentation masks for all objects in the image sample data, and perform test matching between the coordinates of the target object and the bounding boxes corresponding to multiple object segmentation masks to obtain the test matching results. When the test matching result indicates that the coordinates of the target referent are in the target object segmentation mask, the association information between the coordinates of the target referent and the target object segmentation mask is generated. The target object segmentation mask is one of multiple object segmentation masks. Based on image sample data, referential language text information, and the bounding boxes corresponding to the target object segmentation masks, multimodal data samples for the referential object task are obtained.

[0209] In some embodiments, the prediction module 1410 is further configured to: Obtain category names from multiple multimodal data samples, and obtain a global phrase pool based on all category names; For each multimodal data sample, at least one category name that does not exist in the image sample data is selected from the global phrase pool as a negative query sample; Input the multimodal data samples and negative query samples into the initial multimodal language model, and use the autoregressive mechanism of the initial multimodal language model to generate the first training prediction word sequence; The first training prediction word sequence corresponding to the negative query sample is a specific sequence of "coordinates not found".

[0210] In some embodiments, the prediction module 1410 is further configured to: When the second training prediction word sequence is the corresponding target detection box prediction task information, the predicted target detection box is calculated based on the second training prediction word sequence. Based on the intersection-union ratio between the predicted target detection box and the label bounding box corresponding to the coordinates of the label lexical, a set of training reward values ​​corresponding to the second training predicted lexical sequence is obtained; When the second training prediction word sequence is the corresponding target point prediction task information, and the second training prediction word sequence points to an irregular target object, a sample object segmentation mask is generated based on the irregular target object in the corresponding image data sample, and a set of training reward values ​​corresponding to the second training prediction word sequence is obtained based on the matching result between the predicted target point in the second training prediction word sequence and the bounding box corresponding to the sample object segmentation mask. When the second training prediction word sequence is the corresponding target point prediction task information, and the second training prediction word sequence points to a regular target object, the bounding box of the regular target object in the corresponding image data sample is obtained, and based on the matching result of the predicted target point and the bounding box of the regular target object, a set of training reward values ​​corresponding to the second training prediction word sequence is obtained.

[0211] In some embodiments, the prediction module 1410 is further configured to: A set of training reward values ​​is standardized to obtain the relative advantage value within the group; Based on the relative advantage value, the policy gradient method with KL divergence constraint is used to train the fine-tuned multimodal language model through reinforcement learning, thus obtaining the multimodal language model.

[0212] In the above embodiments, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, the specific implementation of the visual task processing device is basically the same as the specific implementation of the visual task processing method described above, and will not be repeated here.

[0213] This application also provides an electronic device, including: At least one memory; At least one processor; At least one program; The program is stored in a memory, and the processor executes the at least one program to implement the visual task processing method described above. The electronic device can be any smart terminal, including mobile phones, tablets, personal digital assistants (PDAs), and in-vehicle computers.

[0214] Please see Figure 15 , Figure 15 The hardware structure of an electronic device according to another embodiment is illustrated. The electronic device includes: The processor 1501 can be implemented using a general-purpose CPU (Central Processing Unit), microprocessor, application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this application. The memory 1502 can be implemented in the form of ROM (Read-Only Memory), static storage device, dynamic storage device, or RAM (Random Access Memory). The memory 1502 can store the operating system and other applications. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program code is stored in the memory 1502 and is called and executed by the processor 1501 using the visual task processing method of the embodiments of this application. The input / output interface 1503 is used to implement information input and output; The communication interface 1504 is used to enable communication and interaction between this device and other devices. Communication can be achieved through wired means (such as USB, Ethernet cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.). Bus 1505 transmits information between various components of the device (e.g., processor 1501, memory 1502, input / output interface 1503, and communication interface 1504); The processor 1501, memory 1502, input / output interface 1503 and communication interface 1504 are connected to each other within the device via bus 1505.

[0215] This application embodiment also provides a storage medium, which is a computer-readable storage medium, storing a computer program that, when executed by a processor, implements the above-described visual task processing method.

[0216] Memory, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs and non-transitory computer-executable programs. Furthermore, memory may include high-speed random access memory, and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, memory may optionally include memory remotely located relative to the processor, and these remote memories can be connected to the processor via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0217] The embodiments described in this application are for the purpose of more clearly illustrating the technical solutions of the embodiments of this application, and do not constitute a limitation on the technical solutions provided by the embodiments of this application. As those skilled in the art will know, with the evolution of technology and the emergence of new application scenarios, the technical solutions provided by the embodiments of this application are also applicable to similar technical problems.

[0218] Those skilled in the art will understand that the technical solutions shown in the figures do not constitute a limitation on the embodiments of this application, and may include more or fewer steps than shown, or combine certain steps, or different steps.

[0219] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0220] Those skilled in the art will understand that all or some of the steps in the methods disclosed above, as well as the functional modules / units in the systems and devices, can be implemented as software, firmware, hardware, or suitable combinations thereof.

[0221] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in the specification and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms “comprising” and “having,” and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0222] It should be understood that in this application, "at least one (item)" means one or more, and "more than" means two or more. "And / or" is used to describe the relationship between related objects, indicating that three relationships can exist. For example, "A and / or B" can represent three cases: only A exists, only B exists, and both A and B exist simultaneously, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one (item) of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one (item) of a, b, or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0223] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of the units described above is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. The coupling or direct coupling or communication connection between the shown or discussed units may be through some interfaces, or indirect coupling or communication connection between the apparatus or units, and may be electrical, mechanical, or other forms.

[0224] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0225] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0226] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes multiple instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing programs, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0227] The preferred embodiments of the present application have been described above with reference to the accompanying drawings, but this does not limit the scope of the claims of the present application. Any modifications, equivalent substitutions, and improvements made by those skilled in the art without departing from the scope and substance of the embodiments of the present application shall be within the scope of the claims of the present application.

Claims

1. A visual task processing method, characterized in that, The method includes: The image and visual task prompt text are input into a multimodal language model for visual task processing, resulting in an output word sequence consisting of target text words and target coordinate words. Decode all the target coordinate words to obtain the target pixel coordinates in the image that correspond to the visual task prompt text; Based on the target pixel coordinates, the target output result corresponding to the visual task prompt text is obtained.

2. The visual task processing method according to claim 1, characterized in that, The multimodal language model includes a visual encoder and a text encoder. The step of inputting images and visual task prompt text into the multimodal language model for visual task processing yields an output word sequence consisting of target text words and target coordinate words, including: The image is input into the visual encoder for visual feature extraction to obtain image features, and the visual task prompt text is input into the text encoder for visual task prompt text word extraction to obtain visual task prompt text words. The image features and the visual task prompt text lexical units are input into the multimodal language model, and the output lexical unit sequence is generated using the autoregressive mechanism of the multimodal language model. The vocabulary of the multimodal language model stores the coordinate encoding mapping relationship between each pixel coordinate in the image and a preset number of special coordinate words.

3. The visual task processing method according to claim 2, characterized in that, The multimodal language model includes an image preprocessing module, which generates the coordinate encoding mapping relationship, including: Based on the size of the image, the coordinates of each pixel in the image are normalized to obtain the corresponding normalized pixel coordinates; Based on a preset number of discrete integers, each normalized pixel coordinate is discretized to obtain a corresponding coordinate quantization value, and each coordinate quantization value corresponds to an image coordinate word. The normalized pixel coordinates are mapped to the image coordinate terms, and the image coordinate terms are added to the vocabulary of the multimodal language model to generate the coordinate encoding mapping relationship. The sequence of two image coordinate terms can be used to characterize a normalized pixel coordinate.

4. The visual task processing method according to claim 3, characterized in that, The step of decoding all the target coordinate words to obtain the target pixel coordinates in the image corresponding to the visual task prompt text includes: Based on the vocabulary, the target coordinate lexical is converted into the corresponding coordinate quantization value; Based on the coordinate encoding mapping relationship, the coordinate quantization value is restored to the target pixel coordinates corresponding to the target coordinate word, and the target pixel coordinates represent the target object region corresponding to the visual task prompt text.

5. The visual task processing method according to claim 1, characterized in that, The target text lexical is used to represent the target text of the visual task, and the target coordinate lexical is used to represent the position of the target object region corresponding to the target text.

6. The visual task processing method according to claim 5, characterized in that, When the visual task is general object detection, the target text word is a category phrase word, and the target coordinate word corresponds to the category phrase word.

7. The visual task processing method according to claim 5, characterized in that, When the visual task is object referential target detection, the target text lexical is a natural language referential sentence lexical, and the target coordinate lexical corresponds to the referential target.

8. The visual task processing method according to claim 5, characterized in that, When the visual task is visual cue target detection, the target text word unit is the visual cue sentence word unit, and the target coordinate word unit corresponds to the cue target.

9. The visual task processing method according to claim 5, characterized in that, When the visual task is text recognition, the target text word is the recognized text word, and the target coordinate word is the coordinate word corresponding to the text region.

10. The visual task processing method according to claim 1, characterized in that, The training steps for the multimodal language model include: Multiple multimodal data samples are acquired, each of which includes image sample data and corresponding label text information and label lexical coordinates; The multimodal data samples are input into the initial multimodal language model, and the first training prediction word sequence is generated using the autoregressive mechanism of the initial multimodal language model. Based on the first training predicted word sequence, the label text information, and the label word coordinates, the cross-entropy loss function is calculated, and the initial multimodal language model is subjected to supervised fine-tuning training to obtain the fine-tuned multimodal language model. At least one sampled training data is obtained from the plurality of multimodal data samples, the sampled training data is input into the fine-tuned multimodal language model, and a second training prediction word sequence is generated using the autoregressive mechanism of the fine-tuned multimodal language model. Based on the second training prediction word sequence, the label text information, and the label word coordinates, a corresponding set of training reward values ​​is calculated. The fine-tuned multimodal language model is trained using reinforcement learning based on the set of training reward values ​​corresponding to the training visual task, thereby obtaining the multimodal language model.

11. The visual task processing method according to claim 10, characterized in that, When the visual task is object detection, the steps for acquiring the multimodal data samples include: Multiple public image data samples are acquired, and the multiple public image data samples are filtered based on a preset resolution and preset compliance constraints to obtain compliant image data samples; Each compliant image data sample is input into the initial multimodal language model to obtain the text description information corresponding to each compliant image data sample; Extract at least one basic category name from each of the text description information, and input each of the compliant image data samples and all the corresponding basic category names into the target detection model for data detection, to obtain the basic detection result of the basic category name in each of the compliant image data samples; The basic detection results are converted to obtain the corresponding tag word coordinates; Based on the compliant image data samples, the basic category names, and the tag lexical coordinates, the multimodal data samples for the target detection task are obtained.

12. The visual task processing method according to claim 10, characterized in that, When the visual task is a referential object task, the steps for acquiring the multimodal sample data include: Acquire initial multimodal sample data and input the initial multimodal sample data into the initial multimodal language model to obtain corresponding referential language text information. The initial multimodal sample data is image sample data including basic bounding boxes and category labels. The textual information of the referential language and the corresponding image sample data are input into a preset target referential model for data processing to obtain the coordinates of the target referential object. Generate object segmentation masks for all objects in the image sample data, and perform test matching between the coordinate points of the target referencing object and the bounding boxes corresponding to multiple object segmentation masks to obtain test matching results; When the test matching result indicates that the coordinates of the target referent are in the target object segmentation mask, the association information between the coordinates of the target referent and the target object segmentation mask is generated, wherein the target object segmentation mask is one of a plurality of object segmentation masks; Based on the image sample data, the referential language text information, and the bounding box corresponding to the target object segmentation mask, the multimodal data sample of the referential object task is obtained.

13. The visual task processing method according to claim 10, characterized in that, The step of inputting the multimodal data samples into an initial multimodal language model and generating a first training prediction word sequence using the autoregressive mechanism of the initial multimodal language model includes: Category names are obtained from multiple multimodal data samples, and a global phrase pool is obtained based on all the category names; For each of the multimodal data samples, at least one category name that does not exist in the image sample data is selected from the global phrase pool as a negative query sample; The multimodal data samples and the negative query samples are input into the initial multimodal language model, and the first training prediction word sequence is generated using the autoregressive mechanism of the initial multimodal language model. The first training prediction word sequence corresponding to the negative query sample is a specific sequence of "coordinates not found".

14. The visual task processing method according to claim 10, characterized in that, The step of calculating a set of training reward values ​​based on the second training predicted word sequence, the label text information, and the label word coordinates includes: When the second training prediction word sequence is the corresponding target detection box prediction task information, the predicted target detection box is calculated based on the second training prediction word sequence. Based on the intersection-union ratio between the predicted target detection box and the label bounding box corresponding to the label lexical coordinates, a set of training reward values ​​corresponding to the second training predicted lexical sequence is obtained; When the second training prediction word sequence is the corresponding target point prediction task information, and the second training prediction word sequence points to an irregular target object, a sample object segmentation mask is generated based on the irregular target object in the corresponding image data sample, and a set of training reward values ​​corresponding to the second training prediction word sequence is obtained based on the matching result between the predicted target point in the second training prediction word sequence and the bounding box corresponding to the sample object segmentation mask. When the second training prediction lexical sequence is the corresponding target point prediction task information, and the second training prediction lexical sequence points to a regular target object, the object bounding box of the regular target object in the corresponding image data sample is obtained, and based on the matching result of the predicted target point and the object bounding box of the regular target object, a set of training reward values ​​corresponding to the second training prediction lexical sequence is obtained.

15. The visual task processing method according to claim 10, characterized in that, The process of training the fine-tuned multimodal language model using reinforcement learning based on the set of training reward values ​​corresponding to the training visual task, to obtain the multimodal language model, includes: The set of training reward values ​​is standardized to obtain the relative advantage value within the group; Based on the relative advantage value, the fine-tuned multimodal language model is trained using the policy gradient method with KL divergence constraints to obtain the multimodal language model.

16. A visual task processing device, characterized in that, The device includes: The prediction module is used to input images and visual task prompt text into a multimodal language model for visual task processing, and to obtain an output word sequence consisting of target text words and target coordinate words; The pixel coordinate conversion module is used to decode all the target coordinate words to obtain the target pixel coordinates in the image that correspond to the visual task prompt text. The result output module is used to obtain the target output result corresponding to the visual task prompt text based on the target pixel coordinates.

17. An electronic device, characterized in that, The electronic device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the visual task processing method according to any one of claims 1 to 15.

18. A storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the visual task processing method according to any one of claims 1 to 15.

Images