Multi-objective large model training method and device, electronic equipment and storage medium

Abstract

The application discloses a multi-target large model training method and device, electronic equipment and storage medium, and relates to the field of image processing. The multi-target large model training method comprises the following steps: constructing a word sequence of real annotation boxes corresponding to multiple target objects in a training image; performing target detection on the training image based on a preset large model, and generating a word sequence of predicted boxes according to the word sequence; constructing a matching cost matrix of the word sequence of the real annotation boxes and the word sequence of the predicted boxes, and obtaining a successfully matched box pair set based on a preset algorithm and the matching cost matrix; constructing a distribution focal loss function based on the successfully matched box pair set, and obtaining a total loss function according to the distribution focal loss function and a preset cross-entropy loss function; and training the training image based on the total loss function. Through the mechanism of "sequence constraint + optimal matching + distribution regression", the application constructs a new training paradigm for autoregressive large model detection, and improves the effectiveness of training.

Description

Technical Field

[0001] This application relates to the technical fields of image processing, and in particular to a method, apparatus, electronic device, and storage medium for training large multi-objective models. Background Technology

[0002] The general object detection technique based on Multimodal Large Language Model (MLLM) mainly adopts the following technical path: After cross-modal alignment and fusion of image features extracted by the visual encoder and instruction text parsed by the text encoder, the continuous two-dimensional coordinate space is discretized into a finite-dimensional set of representable tokens. This allows the model to decode and output the coordinate sequence information of the target bounding box sequentially based on the next token prediction mechanism within a unified autoregressive generation framework. This technical solution has the advantages of simplicity and task uniformity at the theoretical architecture level, that is, it can achieve target localization under open vocabulary conditions without designing a dedicated regression head or classification branch for the detection task.

[0003] However, the above-mentioned technical solutions suffer from the following fundamental technical defects during engineering implementation, which severely restrict their detection performance and practical value: I. Core Architectural Flaw: The Fundamental Conflict Between the Autoregressive Generation Mechanism and the Need for Parallel Detection Existing MLLM-based detectors strictly adhere to the autoregressive decoding paradigm, resulting in a generation process with strong order dependencies—meaning that each time-sequence word prediction depends on all previously generated word sequences and their corresponding attention contexts. This mechanism leads to the serialization and non-parallelization of target bounding box generation. When multiple target instances of the same class exist in an image, the model can only generate bounding box descriptions for each instance sequentially, failing to achieve global parallel prediction like traditional detectors (such as CNN-based single-stage detectors or Transformer-based ensemble prediction detectors). More critically, the training paradigm of these techniques does not impose any explicit constraints on the generation order of similar target boxes, resulting in a random and uncertain sequential pattern in the model output. This deficiency leads to a severe label matching paradox during training. For example, multiple predictions may meet the coordinate accuracy requirements, but due to the inconsistency between the output order and the label order, no loss score can be obtained. Alternatively, the omission / increment of a single instance may cause cascading misalignment of subsequent word sequences, leading to the overall prediction being judged as incorrect, resulting in training signal distortion and optimization direction deviation.

[0004] II. Derivative Technical Issues: Reduced Training Effectiveness and Degraded Detection Performance Due to the aforementioned word order matching problem, the model cannot obtain stable and accurate gradient signals during backpropagation, directly leading to a significant reduction in training effectiveness. Specifically: 1) Low convergence efficiency: The loss function generates a large number of spurious negative samples due to the misalignment of the order, making it difficult for the model to learn a robust space-semantic mapping relationship.

[0005] 2) Limited generalization ability: Even in advanced models such as Qwen2.5-VL and Seed1.5-VL with parameter scales of billions or even tens of billions, there are still common technical defects such as insufficient target localization accuracy, low recall rate, bounding box coordinate drift and repeated prediction.

[0006] 3) Performance gap: In standard target detection benchmark tests such as COCO and LVIS, the overall detection performance of the above models (measured by AP and AR) is still significantly inferior to traditional detectors based on direct coordinate regression, and there is an order of magnitude difference in inference speed.

[0007] In summary, while multimodal large language model detection schemes have theoretical potential in open vocabulary understanding, their inherent lack of sequentiality and insufficient training constraints in their autoregressive generation mechanism constitute structural defects that cannot be overcome by simple parameter expansion or data augmentation. There is an urgent need for an innovative technical solution that can break through the limitations of this paradigm while retaining the advantages of semantic understanding. Summary of the Invention

[0008] Therefore, the purpose of this application is to propose a multi-objective large-scale model training method, apparatus, electronic device, and storage medium. By enforcing a spatially aware serialization strategy to standardize the annotation and generation order, the training data acquires a deterministic order structure, eliminating the need for the model to learn uncertain order relationships. Combined with the Hungarian algorithm, globally optimal matching between predicted bounding boxes and ground truth boxes is achieved, reducing the word-level hard matching error rate from 30-40% in traditional methods to below 5%, ensuring effective gradient feedback for correct predictions, and significantly improving the purity and effectiveness of the training supervision signal.

[0009] This application provides a method for training a large multi-objective model. The method includes: constructing a word sequence of ground truth bounding boxes corresponding to multiple target objects in a training image; performing target detection on the training image based on a preset large model, and generating a word sequence of predicted bounding boxes according to the word sequence; constructing a matching cost matrix of the word sequence of the ground truth bounding boxes and the word sequence of the predicted bounding boxes, and obtaining a set of successfully matched box pairs based on a preset algorithm; constructing a distributed focus loss function based on the set of successfully matched box pairs, and obtaining a total loss function based on the distributed focus loss function and a preset cross-entropy loss function; and training the training image based on the total loss function.

[0010] For example, constructing a lexical sequence of ground truth bounding boxes corresponding to multiple target objects in a training image includes: normalizing the ground truth bounding boxes in the training image to obtain normalized coordinate information of the ground truth bounding boxes; constructing corresponding lexical units for each ground truth bounding box based on the normalized coordinate information of the ground truth bounding box, wherein the lexical units at least represent the diagonal coordinate information of the ground truth bounding box; and sorting the lexical units based on the positional relationship of the ground truth bounding boxes in the training image to obtain a lexical sequence of the ground truth bounding boxes.

[0011] For example, sorting the words based on the positional relationship of the ground truth bounding boxes in the training image includes: sorting the words according to the size relationship of the horizontal coordinates of the ground truth bounding boxes in the training image, and when the horizontal coordinates of the ground truth bounding boxes are the same, sorting them according to the size relationship of the vertical coordinates of the ground truth bounding boxes in the training image, and / or, sorting the words according to the size relationship of the dimensions of the ground truth bounding boxes in the training image.

[0012] For example, constructing the matching cost matrix of the word sequence of the ground truth bounding box and the word sequence of the predicted bounding box includes: determining the negative log probability of the box category prediction of the ground truth bounding box and the predicted bounding box, the L1 regression loss of the ground truth bounding box and the predicted bounding box, and the intersection-union ratio (IUGR) of the ground truth bounding box and the predicted bounding box; determining a first difference between a preset integer and the IUGR, and determining a first product of the first difference and a first hyperparameter; determining a first sum between the first product and the L1 regression loss, and determining a second product of the first sum and a second hyperparameter; determining a third product of the negative log probability and a third hyperparameter, and obtaining the matching cost matrix based on the sum of the second product and the third product.

[0013] For example, before constructing the cost matrix of the word sequence of the ground truth bounding boxes and the word sequence of the predicted bounding boxes, the method further includes: obtaining a first number of ground truth bounding boxes and a second number of predicted bounding boxes; when the second number is greater than the first number, redundant predicted bounding boxes are matched with empty categories; when the first number is greater than the second number, the word sequence of the predicted bounding boxes is filled with dashed boxes to align with the number of word sequences of the ground truth bounding boxes.

[0014] For example, constructing a distribution focus loss function based on the set of successfully matched box pairs includes: for the successfully matched box pairs, converting the word sequence of the ground truth labeled boxes into a narrow variance Gaussian distribution; obtaining the word probability distribution of the predicted boxes, determining the third product of the logarithm of the word probability distribution and a preset power of the word probability distribution, and the second difference between the preset integer and the third product; and determining the product of the second difference and the narrow variance Gaussian distribution as the distribution focus loss function.

[0015] For example, obtaining the total loss function based on the distribution focus loss function and the preset cross-entropy loss function includes: weighting the distribution focus loss function and the preset cross-entropy loss function to obtain the total loss function.

[0016] Another embodiment of this application provides a large-scale multi-objective model training apparatus, the apparatus comprising: a construction module for constructing a word sequence of ground truth bounding boxes corresponding to multiple target objects in a training image; a detection module for performing target detection on the training image based on a preset large-scale model, and generating a word sequence of predicted bounding boxes according to the word sequence; a matching module for constructing a matching cost matrix of the word sequence of the ground truth bounding boxes and the word sequence of the predicted bounding boxes, and obtaining a set of successfully matched box pairs based on a preset algorithm; a loss function module for constructing a distributed focus loss function based on the set of successfully matched box pairs, and obtaining a total loss function based on the distributed focus loss function and a preset cross-entropy loss function; and a training module for training the training image based on the total loss function.

[0017] Another embodiment of this application provides an electronic device, including a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the steps of the above-described multi-objective large model training method.

[0018] Another embodiment of this application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the above-described multi-objective large model training method.

[0019] The above embodiments of the multi-objective large model training method include: constructing a word sequence of ground truth bounding boxes corresponding to multiple target objects in the training image; performing target detection on the training image based on a preset large model, and generating a word sequence of predicted bounding boxes according to the word sequence; constructing a matching cost matrix of the word sequence of ground truth bounding boxes and the word sequence of predicted bounding boxes, and obtaining a set of successfully matched box pairs based on a preset algorithm; constructing a distributed focus loss function based on the set of successfully matched box pairs, and obtaining a total loss function based on the distributed focus loss function and a preset cross-entropy loss function; and training the training image based on the total loss function. This invention constructs a novel training paradigm for autoregressive large model detection through a "sequence constraint + optimal matching + distributed regression" mechanism, improving the effectiveness of training. Attached Figure Description

[0020] Figure 1 A flowchart of a multi-objective large model training method provided for embodiments of this application; Figure 2 A flowchart for constructing a word sequence of real annotation boxes is provided for an embodiment of this application; Figure 3 A schematic diagram of training images provided for embodiments of this application; Figure 4 A flowchart for constructing the matching cost matrix provided in this application embodiment; Figure 5 A flowchart for constructing the distributed focus loss function provided in this application embodiment; Figure 6 A schematic diagram of a multi-objective large model training apparatus provided for an embodiment of this application; Figure 7 A block diagram of an electronic device provided in an embodiment of this application. Detailed Implementation

[0021] The embodiments of this application are described in detail below. Examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and intended to explain this application, and should not be construed as limiting this application.

[0022] Figure 1 This is a flowchart of a multi-objective large model training method according to an embodiment of this application.

[0023] As an example, such as Figure 1 As shown, multi-objective large model training methods include: S101, Construct a sequence of words for the ground truth bounding boxes corresponding to multiple target objects in the training image.

[0024] S102, based on a preset large model, target detection is performed on the training image, and a word sequence of prediction boxes is generated according to the word sequence.

[0025] S103, construct the matching cost matrix of the word sequence of the real labeled box and the word sequence of the predicted box, and obtain the matching cost matrix based on the preset algorithm to obtain the set of successfully matched box pairs.

[0026] S104. Construct a distributed focus loss function based on the set of successfully matched box pairs, and obtain the total loss function based on the distributed focus loss function and the preset cross-entropy loss function.

[0027] S105, train the training images based on the total loss function.

[0028] For example, a training image may include multiple target objects, which can be of the same type or multiple types. The training image includes ground truth bounding boxes corresponding to multiple target objects. It is understood that the overall goal of large model training methods is to make the predicted bounding boxes as similar as possible to the ground truth bounding boxes. However, because there are multiple target objects of the same type in the training image, and traditional labels usually follow random labeling or a simple heuristic spatial order, while the generation order of large models exhibits dynamic uncertainty due to the endogenous randomness of the autoregressive sampling mechanism during inference, this structural asymmetry between the labeling sequence and the generated sequence leads to a fundamental matching paradox during training: even if the model accurately predicts the location and category of all targets, the overall prediction is deemed incorrect simply because the output order is inconsistent with the preset labeling sequence, thus inhibiting the backpropagation of the correct gradient signal. This mismatch problem directly causes distortion of the training supervision signal and low model convergence efficiency, severely hindering the acquisition of robust detection capabilities in multi-target scenarios.

[0029] For example, to address the structural asymmetry between the labeled and generated sequences, this application constructs a word sequence for the ground truth bounding boxes corresponding to multiple target objects in the training image before inputting the training image. It can be understood that the word sequence represents the positional information of each ground truth bounding box in the training image. Based on a pre-defined large model, target detection is performed on the training image, and the word sequence of the predicted bounding boxes is generated in the same way as the above-mentioned method of generating word sequences from ground truth bounding boxes, in order to make the ground truth bounding boxes and predicted bounding boxes structurally symmetrical in sequence as much as possible. However, the specific number of predicted bounding boxes generated is uncertain. For example, objects that are not target objects may be identified as target objects during target detection, and there may be cases of missed identification. Traditional matching techniques use a fixed-position hard matching strategy to align the predicted word sequence with the labeled word sequence position by position, but this mechanism cannot adapt to open scenarios where the number of targets is dynamically variable and the generation order is uncertain. When a single instance of the predicted sequence is missed, redundant, or its order is disturbed, a cascading matching error will be triggered—the correspondence of all subsequent words will shift in a chain reaction, causing a large number of correct localization results to be misjudged as incorrect, resulting in severe noise in the training signal.

[0030] For example, to address the cross-granularity alignment problem between word-level prediction and target-level supervision, this application constructs a matching cost matrix between the word sequences of the ground truth bounding boxes and the word sequences of the predicted bounding boxes before constructing the loss function. The matching cost matrix is ​​then calculated based on a preset algorithm to obtain a set of successfully matched box pairs. It can be understood that this application introduces a one-to-one optimal matching strategy within a set prediction framework to ensure that ground truth bounding boxes and predicted bounding boxes can be matched one-to-one. The preset algorithm can be the Hungarian algorithm. A distributed focus loss function is calculated for the set of successfully matched box pairs, while retaining the traditional cross-entropy loss function. The total loss function is obtained by combining the distributed focus loss function and the cross-entropy loss function, and the training image is trained based on the total loss function.

[0031] This application proposes a novel systematic architecture that, while retaining the powerful semantic understanding capabilities of MLLM, fundamentally overcomes the structural defects of the autoregressive generation paradigm in object detection tasks, completely resolves the training matching paradox, improves the quality of supervision signals, and enforces standardized labeling and generation order through a spatially aware serialization strategy, enabling training data to have a deterministic order structure. The model no longer needs to learn uncertain order relationships. Combined with the Hungarian algorithm, it achieves globally optimal matching between predicted bounding boxes and ground truth labeled boxes, reducing the error rate of word-level hard matching, ensuring that correct predictions obtain effective gradient feedback, and significantly improving the purity and effectiveness of training supervision signals.

[0032] As an example, such as Figure 2 As shown, the word sequence for constructing the ground truth bounding boxes corresponding to multiple target objects in the training image includes: S201, normalize the ground truth bounding boxes in the training image to obtain the normalized coordinate information of the ground truth bounding boxes.

[0033] S202, for each real bounding box, construct the corresponding word element based on the normalized coordinate information of the real bounding box, and the word element at least represents the diagonal coordinate information of the real bounding box.

[0034] S203. Based on the positional relationship of the ground truth bounding boxes in the training image, the word units are sorted to obtain the word unit sequence of the ground truth bounding boxes.

[0035] For example, the ground truth bounding boxes in the training images are first normalized to construct a space: the length and width of any size input image are uniformly mapped to a fixed quantization dimension, such as Q, which can be 1000, forming a normalized virtual coordinate system. Of course, the quantization dimension Q is not limited to 1000 and can be determined as needed. For example, if there are many target objects in the training images, the quantization dimension can be set larger; if there are few target objects in the training images, the quantization dimension can be set smaller.

[0036] For example, the following example illustrates that for a training image with a quantization dimension Q of 1000, the coordinates (x, y) are quantized according to the following formula: x_token = round((x / W) * 999) y_token=round((y / H) * 999) Where x_token represents the normalized x-coordinate, y_token represents the normalized y-coordinate, and round() represents the rounding function.

[0037] For example, for each real bounding box, a corresponding lexical unit is constructed based on the normalized coordinate information of the real bounding box. This application introduces a dedicated lexical unit embedding, reserving a preset number of consecutive special lexical units in the LLM vocabulary to fill in the coordinate information of the real bounding box, such as 1000 special lexical units.<coord_0> to<coord_999> Each lexical uniquely corresponds to a quantized integer value. This lexical set is orthogonal to the model's existing vocabulary to avoid conflicts.

[0038] For example, introducing unlearnable special lexical units <boxstart>(recorded as) <sob>)and <boxend>(recorded as) <eob>() serves as a boundary marker lexical, forming a clear box boundary grammatical structure. This is understandable. <sob>It is the starting word of a real annotation box. <eob>This is its ending term. At least the diagonal coordinates of the true bounding box must be represented between the beginning and ending term. For example, using the top-left and bottom-right two-point representation (x1, y1, x2, y2), after quantization, it is mapped to a sequence of 6 terms: <sob><coord_x1><coord_y1><coord_x2><coord_y2> <eob>.

[0039] For example, for the bounding box (512, 512, 768, 768) in a 2048×1024 image: quantization calculation: x1=250, y1=500, x2=375, y2=750, token sequence: <sob><coord_250><coord_500><coord_375><coord_750> <eob>Each true bounding box corresponds to a sequence of lexical terms.

[0040] For example, to avoid ambiguity in the model output order, this application designs a rigorous space-dominant ranking algorithm. This algorithm ranks the tokens based on the positional relationships of the ground truth bounding boxes in the training image, obtaining a token sequence of the ground truth bounding boxes. In essence, each ground truth bounding box is ranked, and the corresponding token sequences are combined according to the ranking order to obtain a token sequence of all ground truth bounding boxes. This sequence is the label sequence for this training image, which is subsequently used for comparison with the prediction results.

[0041] As an example, based on the positional relationships of the ground truth bounding boxes in the training image, the words are sorted, including: Tokens are sorted according to the magnitude of their horizontal coordinates in the training image, and when the horizontal coordinates of the ground truth bounding boxes are the same, they are sorted according to the magnitude of their vertical coordinates in the training image, and / or... The words are sorted according to the size relationship of the true bounding boxes in the training images.

[0042] For example, the boxes can be sorted according to the size of their horizontal coordinates in the training image, such as from smallest to largest or from largest to smallest. When the horizontal coordinates of the boxes are the same, the vertical coordinates are then compared. The vertical coordinates can be sorted from smallest to largest or from largest to smallest.

[0043] For example, the quantized top-left corner coordinates (x1_token, y1_token) of each box are used as the primary sorting key. The hierarchical sorting rules are: primary order: sort by x1_token from smallest to largest (from left to right); secondary order: when x1_token are the same, sort by y1_token from smallest to largest (from top to bottom).

[0044] For example, the words can also be sorted according to the size relationship of the real bounding boxes in the training image. For example, based on the coordinate information of the real bounding boxes, the area ratio of the real bounding boxes to the training image can be obtained, and the words can be sorted according to the size relationship of the real bounding boxes, for example, from largest to smallest or from smallest to largest.

[0045] For example, such as Figure 3 The image shown is a schematic diagram of a training image, which is a 1280*1280 pixel image. There are three coordinate boxes labeled "dog", represented by the top left and bottom right points, which are (146, 456, 504, 1012), (412, 545, 742, 1065), and (845, 451, 1143, 1160).

[0046] The coordinate frame is normalized to (0, 999) using the method described above, and then sorted according to spatial information. The processed training data format is as follows, where... <sob>and <eos>These are the paragraph start and end markers: the coordinate frame of the dog in the image is... <sob> <114><356><394><790> <eob> <sob> <322><425><579><831> <eob> <sob> <660><352><892><905> <eob> <eos>Following this method, all image and coordinate frame data are organized in this format.

[0047] During the training data construction phase, the above sorting is performed on all ground truth (GT) boxes for each image to generate a deterministic target sequence. During inference, the model is required to generate the box sequence in this order to ensure that the output is structurally aligned with the spatial layout of the image.

[0048] This application uses spatial ordering rules as constraints during inference to eliminate the uncertainty of the order in the inference stage, ensuring that the output order of the model is strictly consistent with the spatial layout of the image, which facilitates the analysis and visualization of downstream tasks. This feature enables the solution of this application to be seamlessly integrated into existing video analysis pipelines and supports stable frame-by-frame tracking.

[0049] To overcome the inherent defects of traditional word-by-word meta-cross-entropy (CE) loss in box-level supervision—that is, ignoring the overall correspondence and structural integrity between predicted boxes and ground truth boxes (GT)—this application introduces a one-to-one optimal matching strategy under the set prediction framework.

[0050] As an example, such as Figure 4 As shown, the matching cost matrix for constructing the word sequence of the ground truth bounding box and the word sequence of the predicted bounding box includes: S401, determine the negative log probability of the box category prediction for the ground truth bounding box and the predicted bounding box, the L1 regression loss for the ground truth bounding box and the predicted bounding box, and the crossover union ratio (CUB) value for the ground truth bounding box and the predicted bounding box.

[0051] S402, determine the first difference between the preset integer and the intersection-union ratio value, and determine the first product of the first difference and the first hyperparameter.

[0052] S403, determine the first summation between the first product and the L1 regression loss value, and determine the second product between the first summation and the second hyperparameter.

[0053] S404 determines the third product of the negative log probability and the third hyperparameter, and obtains the matching cost matrix based on the sum of the second and third products.

[0054] For example, the sequence of boxes generated by the autoregression of a large model is regarded as an unordered set B = {b1, b2, ..., b}. n }, where each element b i This is the parsed structured predicted bounding box object. The set of ground truth bounding boxes is denoted as G = {g1,...,g...}. m The training objective is transformed into finding the minimum-cost bipartite graph matching from B to G, rather than forcing sequence position alignment.

[0055] For example, in each training forward propagation, the n×m dimensional matching cost matrix C is dynamically calculated, and its elements C ij It consists of the following weighted terms:

[0056] in, Let represent the elements of the i-th predicted bounding box and the j-th ground truth bounding box in the matching cost matrix C. The negative log probability of the bounding box category (if the dataset contains category labels). The L1 regression loss is applied to the embedding vectors corresponding to the four coordinate words. The IoU is the intersection-union ratio of the ground truth bounding box and the predicted bounding box, which is used to strengthen the spatial overlap constraint. For the i-th predicted box, For the j-th true bounding box, Let i be the category of the i-th predicted box. Let j be the category of the j-th real bounding box. As the first hyperparameter, This is the second hyperparameter. This is the second hyperparameter. The default integer is 1, and the first difference between integer 1 and the intersection-union ratio cost is determined. Let the first hyperparameter be . It can be 2.0. Determine the first product. L1 regression loss value The first sum between Let the second hyperparameter be... The product can be 1.0, and the second product is... Let the third hyperparameter be... It can be 0.5. The product of the negative logarithmic probability and the third hyperparameter is... Finally, the sum of the second and third products yields the matching cost matrix. .

[0057] For example, the Hungarian algorithm can be used to solve for the minimum weight matching in polynomial time to obtain the optimal assignment. ,in, This represents the optimal matching mapping relationship. This indicates the mapping relationship between successfully matched predicted bounding boxes and ground truth bounding boxes. Only for successfully matched predicted bounding box pairs (… ) Calculate the subsequent loss; unmatched predicted boxes are classified as negative samples, and only a class confidence penalty is applied. Wherein, This represents the predicted bounding box in the i-th predicted bounding box and the j-th ground truth bounding box that were successfully matched. This represents the j-th ground truth bounding box and the i-th predicted bounding box that were successfully matched.

[0058] Before calculating the loss function, the present application matches the prediction box sequence and the real label box sequence, and calculates the loss function for the successfully matched box pairs, which can adapt to open scenarios with dynamically variable target quantities and uncertain generation orders. When a single instance is missing, redundant, or has its order perturbed in the prediction sequence, the correspondence of other tokens will not be affected.

[0059] As an example, before constructing the cost matrix of the token sequence of the real annotation boxes and the token sequence of the prediction boxes, the multi-target large model training method further comprises: acquiring a first quantity of the real annotation boxes and a second quantity of the prediction boxes; when the second quantity is greater than the first quantity, matching the redundant prediction boxes with the empty category; when the first quantity is greater than the second quantity, padding the token sequence of the prediction boxes with dummy boxes to align the quantity with the token sequence of the real annotation boxes.

[0060] Exemplarily, the quantity of the real annotation boxes and the quantity of the prediction boxes may not necessarily completely match. For example, in target detection, an object that is not a target object may be recognized as a target object, or missed recognition and other situations may occur. Of course, it is expected in model training that the two are completely equal. The present application proposes a dynamic box quantity processing method, which automatically supplements the defective box quantity when the quantity of the real annotation boxes does not match the quantity of the prediction boxes, so as to satisfy the calculation of the cost matrix.

[0061] Exemplarily, let the first quantity of the real annotation boxes be m and the second quantity of the prediction boxes be n. When the prediction box quantity n is greater than the GT quantity m, the redundant prediction boxes are automatically matched to the "empty" category. When n is less than m, the prediction boxes are supplemented by padding "dummy boxes" to ensure that the algorithm can run. This mechanism naturally supports variable-quantity target detection without requiring non-maximum suppression (NMS) post-processing.

[0062] The matching mechanism of the present application frees the model from dependence on the positions of the prediction sequence, allows flexible box order output, significantly reduces gradient conflicts in the initial stage of training, and improves the positioning robustness in occluded and dense scenarios.

[0063] As an example, as Figure 5 shown, constructing a distribution focal loss function based on the successfully matched box pair set comprises: S501, for the successfully matched box pairs, converting the token sequence of the real annotation boxes into a narrow-variance Gaussian distribution.

[0064] S502, acquiring the token probability distribution of the prediction boxes, and determining a third product of the logarithm of the token probability distribution and a preset power of the token probability distribution, as well as a second difference between a preset integer and the third product.

[0065] S503, determine the product of the second difference and the narrow variance Gaussian distribution as the distribution focus loss function.

[0066] For example, given coordinate lexical units<coord_0> to<coord_999> Due to its inherent ordinal property, the traditional one-hot CE loss (cross-entropy loss) forces the model to learn hard classification, leading to adjacent coordinates (such as...)<coord_499> and<coord_500> In the embedding space, coordinates are incorrectly treated as completely independent categories, ignoring the prior knowledge that "spatial proximity equals semantic similarity." To address this, this application introduces Distribution Focal Loss (DFL) to reconstruct coordinate prediction as a probability distribution regression problem.

[0067] For example, for successfully matched bounding box pairs, the word sequence of the ground truth bounding boxes is transformed into a narrow variance Gaussian distribution, that is, the ground truth coordinates v∈[0, 999] are converted into a narrow variance Gaussian distribution as soft labels, as shown below:

[0068] in, This represents the label of the k-th word, where k is the independent variable and its value ranges from [0, 999]. This represents the mean of a Gaussian distribution. The calculation iterates through all true coordinates v. This represents the standard deviation of the Gaussian distribution, used to control the smoothing intensity, and is typically set to 1.0~2.0. This design makes the nearest neighbor... The lexical units obtain non-zero supervision signals, forcing the embedding space to maintain the ordinal topology.

[0069] For example, the word probability distribution of the predicted bounding box is obtained, and a distribution focus loss function is constructed based on the word probability distribution of the predicted bounding box. The distribution can be weighted using the form of focal loss to focus on the coordinates of hard instances. The distribution focus loss function is shown below:

[0070] in, Represents the distribution focus loss function. To focus the parameters (usually 2), effectively suppress the gradient contribution of easily classified words and accelerate convergence. P(k) is the word probability distribution of the predicted box. This loss makes the Euclidean distance between adjacent coordinate words in the embedding space closer, forming a continuous semantic manifold, and further improving the pixel-level accuracy of small target localization.

[0071] As an example, the total loss function is obtained based on the distributed focus loss function and the preset cross-entropy loss function, including: weighting the distributed focus loss function and the preset cross-entropy loss function to obtain the total loss function.

[0072] For example, by weighting the distribution focus loss function and the preset cross-entropy loss function, the total loss of the coordinate box in a single training iteration is:

[0073] Where M is the set of successfully matched box pairs. For the i-th prediction box and the j-th true bounding box The distribution focus loss function, The weights of the distribution focus loss function, To pre-define the weights of the cross-entropy loss function, The cross-entropy loss function represents SOB terms. The cross-entropy loss function represents the EOB term. This loss design achieves synergistic optimization of box-level global optima and coordinate-level semantic smoothness, enabling the model to maintain the efficiency of discrete term generation while gaining the accuracy advantage of continuous space regression.

[0074] It should be noted that all normal words are still calculated using the CE loss function, while the newly designed coordinate words are not included in the CE loss calculation; instead, the loss is calculated using the distributed focus loss after Hungarian matching mentioned in this method. Finally, the model is trained on the training image set based on the total loss function to obtain various optimized hyperparameters.

[0075] This application utilizes DFL loss and embedding regularization to enable the model to learn the geometric continuity of ordinal terms, thus exhibiting stronger robustness in cross-resolution tests.

[0076] For example, with Figure 3 The training image shown is used as an example. During training, if the model's output is the bounding box of the dog in the image, it would be: <sob> <118><356><394><790> <eob> <sob> <660><352><892><905> <eob> <eos>.

[0077] Based on Hungarian matching, the two coordinate boxes will be matched to the first and third coordinate boxes in the label, respectively, for calculation. Among them, word elements <118> Loss is incurred where the predicted result differs from the label, resulting in gradient backpropagation. In this example, because one word is missing from the predicted result, the loss occurs. <eos>This will result in significant losses because the sentence shouldn't be finished yet, thus generating gradients and supervising model learning. Similarly, if there are too many predictions, the corresponding coordinate boxes will suffer greater losses. <sob> <eob>The model will suffer losses, namely, words that should not appear, thereby supervising the model's learning to reduce false positives.

[0078] The multi-objective large model training method of this application is compatible with mainstream multimodal large models and can be directly embedded. Other configurations and training methods are consistent with multimodal large model training methods. For example, it can be fine-tuned based on the QWEN series models to improve the model's visual localization and detection capabilities.

[0079] The multi-objective large model training method of this application constructs a new training paradigm for autoregressive MLLM detection through a triple architecture of "spatial order constraint + optimal matching + distribution regression". It achieves breakthrough improvements in three dimensions: detection accuracy, convergence speed, and engineering deployment, paving the way for the practical application of general visual understanding models.

[0080] This application also proposes a multi-objective large model training device.

[0081] As an example, such as Figure 6 As shown, the multi-object large model training device includes: a construction module 601, used to construct the word sequence of ground truth bounding boxes corresponding to multiple target objects in the training image; a detection module 602, used to perform target detection on the training image based on a preset large model, and generate the word sequence of predicted boxes according to the word sequence; a matching module 603, used to construct a matching cost matrix of the word sequence of ground truth bounding boxes and the word sequence of predicted boxes, and obtain the matching cost matrix based on a preset algorithm to obtain a set of successfully matched box pairs; a loss function module 604, used to construct a distributed focus loss function based on the set of successfully matched box pairs, and obtain a total loss function based on the distributed focus loss function and a preset cross-entropy loss function; and a training module 605, used to train the training image based on the total loss function.

[0082] This application also proposes a computer-readable storage medium.

[0083] In this embodiment, a computer program is stored on a computer-readable storage medium, and when the computer program is executed by a processor, it implements the steps of the above-described multi-objective large model training method.

[0084] Figure 7 A block diagram of an electronic device provided in an embodiment of this application.

[0085] This application provides an electronic device, including a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the above-mentioned multi-objective large model training method.

[0086] like Figure 7 As shown, for ease of understanding, embodiments of this application illustrate a specific electronic device.

[0087] Electronic devices are intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. Electronic devices can also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0088] like Figure 7 As shown, the device includes a computing unit 701, which can perform various appropriate actions and processes based on a computer program stored in a read-only memory (ROM) 702 or a computer program loaded into a random access memory (RAM) 703 from a storage unit 708. The RAM 703 may also store various programs and data required for the operation of the electronic device. The computing unit 701, ROM 702, and RAM 703 are interconnected via a bus 704. An input / output (I / O) interface 705 is also connected to the bus 704.

[0089] Multiple components in the electronic device are connected to the I / O interface 705. These components include: an input unit 706, such as a keyboard or mouse; an output unit 707, such as various types of displays or speakers; a storage unit 708, such as a disk or optical disk; and a communication unit 709, such as a network interface card (NIC), a modem, or a wireless transceiver. The communication unit 709 allows the electronic device to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0090] The computing unit 701 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 701 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 701 performs the various methods described above, such as multi-objective large model training methods. For example, in some embodiments, the multi-objective large model training method can be implemented as a computer software program tangibly contained in a machine-readable medium, such as storage unit 708. In some embodiments, part or all of the computer program can be loaded and / or installed on an electronic device via ROM 702 and / or communication unit 709. When the computer program is loaded into RAM 703 and executed by the computing unit 701, the multi-objective large model training method described above can be executed. Alternatively, in other embodiments, the computing unit 701 can be configured to perform multi-objective large model training methods by any other suitable means (e.g., by means of firmware).

[0091] It should be noted that the logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be specifically implemented in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (such as a computer-based system, a processor-included system, or other system that can fetch and execute instructions from, an instruction execution system, apparatus, or device). For the purposes of this application, "computer-readable medium" can be any means that can contain, store, communicate, propagate, or transmit programs for use by, or in conjunction with, an instruction execution system, apparatus, or device. More specific examples (a non-exhaustive list) of computer-readable media include: electrical connections (electronic devices) having one or more wires, portable computer disk drives (magnetic devices), random access memory (RAM), read-only memory (ROM), erasable and editable read-only memory (EPROM or flash memory), fiber optic devices, and portable optical disc read-only memory (CDROM). Furthermore, computer-readable media can even be paper or other suitable media on which programs can be printed, because programs can be obtained electronically, for example, by optically scanning the paper or other media, followed by editing, interpreting, or otherwise processing as necessary, and then stored in computer memory.

[0092] It should be understood that various parts of this application can be implemented using hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented using software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0093] In the description of this application, the references to "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of this application. In this application, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0094] In the description of this application, it should be understood that the terms "center", "longitudinal", "lateral", "length", "width", "thickness", "upper", "lower", "front", "rear", "left", "right", "vertical", "horizontal", "top", "bottom", "inner", "outer", "clockwise", "counterclockwise", "axial", "radial", "circumferential", etc., indicating the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings, are only for the convenience of describing this application and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation, and therefore should not be construed as a limitation of this application.

[0095] Furthermore, the terms "first," "second," etc., used in the embodiments of this application are for descriptive purposes only and should not be construed as indicating or implying relative importance, or implicitly specifying the number of technical features indicated in this embodiment. Therefore, features defined with terms such as "first" and "second" in the embodiments of this application can explicitly or implicitly indicate that the embodiment includes at least one of those features. In the description of this application, the word "multiple" means at least two or more, such as two, three, four, etc., unless otherwise explicitly specified in the embodiments.

[0096] In this application, unless otherwise explicitly specified or limited in the embodiments, the terms "installation," "connection," "joining," and "fixing" appearing in the embodiments should be interpreted broadly. For example, a connection can be a fixed connection, a detachable connection, or an integral part; it can also be a mechanical connection, an electrical connection, etc. Of course, it can also be a direct connection, or an indirect connection through an intermediate medium, or it can be the internal communication between two components, or the interaction between two components. Those skilled in the art can understand the specific meaning of the above terms in this application based on the specific implementation.

[0097] In this application, unless otherwise expressly specified and limited, "above" or "below" the second feature can mean that the first and second features are in direct contact, or that the first and second features are in indirect contact through an intermediate medium. Furthermore, "above," "on top of," and "over" the second feature can mean that the first feature is directly above or diagonally above the second feature, or simply that the first feature is at a higher horizontal level than the second feature. "Below," "below," and "under" the second feature can mean that the first feature is directly below or diagonally below the second feature, or simply that the first feature is at a lower horizontal level than the second feature.

[0098] Although embodiments of this application have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting this application. Those skilled in the art can make changes, modifications, substitutions and variations to the above embodiments within the scope of this application.< / eob> < / sob> < / eos> < / eos> < / eob> < / sob> < / eob> < / sob> < / eos> < / eob> < / sob> < / eob> < / sob> < / eob> < / sob> < / eos> < / sob> < / eob> < / sob> < / eob> < / sob> < / eob> < / sob> < / eob> < / boxend> < / sob> < / boxstart>

Claims

1. A method for training large multi-objective models, characterized in that, The method includes: Construct a word sequence of ground truth bounding boxes corresponding to multiple target objects in the training image; The training images are used to perform object detection based on a pre-defined large model, and a word sequence of prediction boxes is generated according to the word sequence. Construct a matching cost matrix between the word sequence of the real labeled bounding box and the word sequence of the predicted bounding box, and obtain the matching cost matrix based on a preset algorithm to get a set of successfully matched box pairs; A distributed focus loss function is constructed based on the set of successfully matched box pairs, and the total loss function is obtained according to the distributed focus loss function and the preset cross-entropy loss function. The training images are trained based on the total loss function; The step of constructing the word sequence of ground truth bounding boxes corresponding to multiple target objects in the training image includes: The ground truth bounding boxes in the training image are normalized to obtain the normalized coordinate information of the ground truth bounding boxes; For each of the ground truth bounding boxes, a corresponding word element is constructed based on the normalized coordinate information and boundary words of the ground truth bounding box. The word element at least represents the diagonal coordinate information of the ground truth bounding box. Based on the positional relationship of the ground truth bounding boxes in the training image, the words are sorted to obtain the word sequence of the ground truth bounding boxes.

2. The multi-objective large model training method according to claim 1, characterized in that, The step of sorting the lexical units based on the positional relationships of the ground truth bounding boxes in the training image includes: The words are sorted according to the size of the horizontal coordinates of the ground truth bounding boxes in the training image, and when the horizontal coordinates of the ground truth bounding boxes are the same, they are sorted according to the size of their vertical coordinates in the training image, and / or, The lexical units are sorted according to the size relationship of the true labeled boxes in the training image.

3. The multi-objective large model training method according to claim 1, characterized in that, The construction of the matching cost matrix for the word sequence of the ground truth bounding box and the word sequence of the predicted bounding box includes: Determine the negative log probability values ​​of the box category predictions for the ground truth bounding boxes and the predicted bounding boxes, the L1 regression loss values ​​for the ground truth bounding boxes and the predicted bounding boxes, and the crossover union ratio (CUB) values ​​for the ground truth bounding boxes and the predicted bounding boxes. Determine a first difference between a preset integer and the intersection-union ratio replacement value, and determine a first product of the first difference and a first hyperparameter; Determine a first sum between the first product and the L1 regression loss value, and determine a second product between the first sum and the second hyperparameter; The third product of the negative log probability and the third hyperparameter is determined, and the matching cost matrix is ​​obtained based on the sum of the second product and the third product.

4. The multi-objective large model training method according to claim 1, characterized in that, Before constructing the matching cost matrix of the word sequence of the ground truth bounding box and the word sequence of the predicted bounding box, the method further includes: Obtain the first number of the true labeled boxes and the second number of the predicted boxes; When the second quantity is greater than the first quantity, the redundant prediction box is matched with the empty category; When the first number is greater than the second number, the word sequence of the predicted box is filled with a dashed box to align with the number of word sequences in the true labeled box.

5. The multi-objective large model training method according to claim 1, characterized in that, The construction of the distributed focus loss function based on the set of successfully matched box pairs includes: For the successfully matched box pairs, the word sequence of the ground truth bounding boxes is transformed into a narrow variance Gaussian distribution; Obtain the word probability distribution of the prediction box, determine the third product of the logarithm of the word probability distribution and the preset power of the word probability distribution, and the second difference between the preset integer and the third product; The product of the second difference and the narrow variance Gaussian distribution is determined as the distribution focus loss function.

6. The multi-objective large model training method according to claim 1, characterized in that, The step of obtaining the total loss function based on the distribution focus loss function and the preset cross-entropy loss function includes: The distribution focus loss function and the preset cross-entropy loss function are weighted to obtain the total loss function.

7. A multi-objective large model training device, characterized in that, The device includes: The building module is used to construct a sequence of words for the ground truth bounding boxes corresponding to multiple target objects in the training image; The detection module is used to perform target detection on the training image based on a preset large model, and generate a word sequence of prediction boxes according to the word sequence; The matching module is used to construct a matching cost matrix between the word sequence of the real labeled box and the word sequence of the predicted box, and to obtain the matching cost matrix based on a preset algorithm to obtain a set of successfully matched box pairs. The loss function module is used to construct a distributed focus loss function based on the set of successfully matched box pairs, and to obtain the total loss function based on the distributed focus loss function and the preset cross-entropy loss function. The training module is used to train the training images based on the total loss function; The construction module is further configured to: normalize the ground truth bounding boxes in the training image to obtain normalized coordinate information of the ground truth bounding boxes; for each ground truth bounding box, construct corresponding lexical units based on the normalized coordinate information of the ground truth bounding box and boundary lexical units, wherein the lexical units at least represent the diagonal coordinate information of the ground truth bounding box; and sort the lexical units based on the positional relationship of the ground truth bounding boxes in the training image to obtain a lexical sequence of the ground truth bounding boxes.

8. An electronic device, characterized in that, It includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the steps of the multi-objective large model training method according to any one of claims 1-6.

9. A computer-readable storage medium, characterized in that, It stores a computer program that, when executed by a processor, implements the steps of the multi-objective large model training method according to any one of claims 1-6.

Images