Model training methods
By filtering and deleting low-quality pseudo-labels during multimodal model training, a target pseudo-label set is generated, which solves the problem of low detection accuracy during multimodal model training and improves the model's detection accuracy.
Patent Information
- Application Number
- CN202611161559.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2026-07-31
- Publication Date
- 2026-11-17
- Estimated Expiration
- 2046-07-31
AI Technical Summary
In multimodal model training, there is a problem of low detection accuracy due to false label noise.
By obtaining an initial set of pseudo-labels for training images, low-quality pseudo-labels are filtered and deleted based on the multi-dimensional label information of the pseudo-labels, a target set of pseudo-labels is generated, and the neural network model is trained using the target set of pseudo-labels.
It improves the detection accuracy of the model and reduces the accumulation of errors caused by noise in the training samples during the model training process.
Smart Images

Figure CN122759262B_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of computer technology, and in particular to model training methods. Background Technology
[0002] With the rapid development of large-scale multimodal models in fields such as image understanding and cross-modal interaction, their application potential in object detection tasks has attracted widespread attention and has gradually penetrated into multiple core scenarios such as autonomous driving, intelligent monitoring, robot vision, and industrial quality inspection. However, the object detection datasets on which multimodal model training depends generally suffer from problems such as incomplete annotation and scarcity of long-tail category samples. It is necessary to expand the data distribution and make up for the annotation defects through pseudo-label technology. However, the initial pseudo-labels generated directly by professional detectors contain a lot of noise, which will inversely affect the model training accuracy, resulting in low detection accuracy of the trained model. Summary of the Invention
[0003] This application provides a model training method to at least address the problem of low detection accuracy of models in related technologies.
[0004] This application provides a model training method, comprising: obtaining an initial pseudo-label set for each of N training images to obtain N initial pseudo-label sets, wherein the i-th initial pseudo-label set includes pseudo-labels obtained by detecting objects in the i-th training image, each pseudo-label in the i-th initial pseudo-label set is used to identify an object in the i-th training image, the N initial pseudo-label sets include the i-th initial pseudo-label set, the N training images include the i-th training image, N is an integer greater than 0, and i is an integer greater than 0 and less than or equal to N; based on the label information included in the pseudo-labels in the N initial pseudo-label sets, deleting at least a portion of the pseudo-labels in at least one of the N initial pseudo-label sets to obtain N target pseudo-label sets; and training a neural network model to be trained based on the N target pseudo-label sets and the N training images to obtain a target neural network model.
[0005] This application also provides a model training apparatus, comprising: an acquisition module, configured to acquire initial pseudo-label sets for each of N training images to obtain N initial pseudo-label sets, wherein the i-th initial pseudo-label set includes pseudo-labels obtained by detecting objects in the i-th training image, each pseudo-label in the i-th initial pseudo-label set is used to identify an object in the i-th training image, the N initial pseudo-label sets include the i-th initial pseudo-label set, the N training images include the i-th training image, N is an integer greater than 0, and i is an integer greater than 0 and less than or equal to N; a deletion module, configured to delete at least a portion of the pseudo-labels in at least one of the N initial pseudo-label sets based on the label information included in the pseudo-labels in the N initial pseudo-label sets to obtain N target pseudo-label sets; and a training module, configured to train a neural network model to be trained based on the N target pseudo-label sets and the N training images to obtain a target neural network model.
[0006] This application also provides an electronic device, including: a memory for storing a computer program; and a processor for executing the computer program to implement the steps of any of the above-described model training methods.
[0007] This application also provides a computer-readable storage medium storing a computer program, wherein the computer program, when executed by a processor, implements the steps of any of the above-described model training methods.
[0008] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of any of the above-described model training methods.
[0009] This application involves detecting objects in each of N training images to generate pseudo-labels for identifying objects in each training image, resulting in N initial pseudo-label sets for the N training images. Then, based on the label information included in each pseudo-label in the N initial pseudo-label sets, the pseudo-labels in the N initial pseudo-label sets are filtered, and at least a portion of the pseudo-labels in at least one of the N initial pseudo-label sets are deleted, resulting in N target pseudo-label sets for the N training images. Finally, a neural network model to be trained is trained based on the N target pseudo-label sets and the N training images to obtain the target neural network model.
[0010] By filtering the generated pseudo-labels through multi-dimensional label information included in the pseudo-labels, and dynamically deleting at least some pseudo-labels from the pseudo-label set, low-quality pseudo-labels can be filtered out. This helps to retain effective supplementary samples while avoiding the introduction of noise, reducing the accumulation of model errors caused by training sample noise during model training, improving the model's training accuracy, and enhancing the model's detection accuracy. Therefore, it can solve the technical problem of low detection accuracy in related technologies and achieve the technical effect of improving the model's detection accuracy. Attached Figure Description
[0011] To more clearly illustrate the embodiments of this application, the accompanying drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0012] Figure 1 This is a hardware structure block diagram of the model training method according to an embodiment of this application;
[0013] Figure 2 This is a flowchart of a model training method according to an embodiment of this application;
[0014] Figure 3 This is a schematic diagram of a pseudo-label according to an embodiment of this application;
[0015] Figure 4 This is a schematic diagram of a real label according to an embodiment of this application;
[0016] Figure 5 This is a schematic diagram of training labels according to an embodiment of this application;
[0017] Figure 6 This is a flowchart of model training according to an embodiment of this application;
[0018] Figure 7 This is a structural block diagram of a model training device according to an embodiment of this application. Detailed Implementation
[0019] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the protection scope of this application.
[0020] It should be noted that, in the description of this application, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. The terms "first," "second," etc., in this application are used to distinguish similar objects and are not used to describe a specific order or sequence.
[0021] To enable those skilled in the art to better understand the present application, the present application will be further described in detail below with reference to the accompanying drawings and specific embodiments.
[0022] The specific application environment architecture or specific hardware architecture on which the execution of the model training method depends is described here.
[0023] The methods and embodiments provided in this application can be executed on a server device or a similar computing device. Taking running on a server device as an example, Figure 1 This is a hardware structure block diagram of the model training method according to an embodiment of this application. For example... Figure 1 As shown, the server device may include one or more ( Figure 1 Only one is shown in the image. A processor 102 (which may include, but is not limited to, a central processing unit (CPU), microprocessor (MCU), or programmable logic device (FPGA), etc.) and a memory 104 for storing data are also shown. The server device may further include a transmission device 106 for communication functions and an input / output device 108. Those skilled in the art will understand that... Figure 1 The structure shown is for illustrative purposes only and does not limit the structure of the server equipment described above. For example, the server equipment may also include components that are more... Figure 1 The more or fewer components shown, or having the same Figure 1 The different configurations shown.
[0024] The memory 104 can be used to store computer programs, such as application software programs and modules, like the computer program corresponding to the model training method in this embodiment. The processor 102 executes various functional applications and data processing by running the computer program stored in the memory 104, thus implementing the above-described method. The memory 104 may include high-speed random access memory and non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 104 may further include memory remotely located relative to the processor 102, and these remote memories can be connected to server devices via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.
[0025] The transmission device 106 is used to receive or send data via a network. Specific examples of the network described above may include a wireless network provided by a communication provider for the server device. In one example, the transmission device 106 includes a Network Interface Controller (NIC), which can connect to other network devices via a base station to communicate with the Internet. In another example, the transmission device 106 may be a Radio Frequency (RF) module used for wireless communication with the Internet.
[0026] The embodiments of this application provide a model training method, and the method is described in detail in conjunction with the execution flow of the model training method.
[0027] This embodiment provides a model training method. Figure 2 This is a flowchart of a model training method according to an embodiment of this application, such as... Figure 2 As shown, the method includes the following steps:
[0028] Step S202: Obtain the initial pseudo-label set of each training image in N training images, resulting in N initial pseudo-label sets. The i-th initial pseudo-label set includes pseudo-labels obtained by detecting objects in the i-th training image. Each pseudo-label in the i-th initial pseudo-label set is used to identify an object in the i-th training image. The N initial pseudo-label sets include the i-th initial pseudo-label set, and the N training images include the i-th training image. N is an integer greater than 0, and i is an integer greater than 0 and less than or equal to N.
[0029] The training images mentioned above can be images used to train the model to be trained (such as a neural network model), and these images can be remote sensing images, medical images, natural images (such as photographs), etc. Each training image includes at least one object to be identified (i.e., an object).
[0030] The aforementioned pseudo-labels can be labels output by a professional detector after detecting and recognizing the training images, used to identify at least one object included in the training images, and the category of each object. In the embodiments of this application, pseudo-labels can be represented by a triplet data, such as ( , , ),in, This represents the coordinates of the bounding box (i.e., the object detection box, the rectangular region used to select the j-th object) in the training image. The coordinates of the bounding box can be represented by the coordinates of the top-left and bottom-right vertices of the bounding box. This indicates that a professional detector has identified the category label of the j-th object in the training image. This represents the confidence level of the professional detector output, which is also the confidence level of the pseudo-label used to identify the j-th object in the training image. The confidence level ranges from [0,1].
[0031] In this embodiment, before training the model to be trained, N training images are first acquired. Then, a professional detector is used to detect and identify each of the N training images, generating a pseudo-label for each training image, resulting in N initial pseudo-label sets for the N training images. Each initial pseudo-label set corresponds to one training image, and each initial pseudo-label set includes at least one pseudo-label, with each pseudo-label used to identify an object in a training image.
[0032] Example, Figure 3 This is a schematic diagram of a pseudo-label according to an embodiment of this application, such as... Figure 3 As shown, the training image includes seven objects (two triangles, two stars, and three circles). A specialized detector is used to detect and identify these objects, generating a pseudo-label for each object (e.g., ...). Figure 3 As shown in the dashed box in the middle, the initial pseudo-label set of the training image is obtained. At this time, the initial pseudo-label set of the training image includes 7 pseudo-labels, namely {pseudo-label 1, pseudo-label 2, ..., pseudo-label 7}. Each pseudo-label is represented by a triplet data, such as pseudo-label 1 = ([x1,y1,x2,y2], triangle, confidence 1), pseudo-label 2 = ([x3,y3,x4,y4], circle, confidence 2), etc.
[0033] Step S204: Based on the tag information included in the pseudo-tags in the N initial pseudo-tag sets, delete at least a portion of the pseudo-tags in at least one of the N initial pseudo-tag sets to obtain N target pseudo-tag sets.
[0034] The aforementioned label information can be used to describe pseudo-labels, and this information may include, but is not limited to, bounding box coordinates, category labels, confidence scores, probability distributions, and other label-related data. For example, ... Figure 3 The pseudo-label 1 in the example, which is defined as ([x1,y1,x2,y2], triangle, confidence 1), can include the label information of [x1,y1,x2,y2] (i.e., the coordinates of the bounding box, which is also the coordinates of the rectangular area corresponding to pseudo-label 1), triangle (i.e., category label), confidence 1 (i.e., the confidence of pseudo-label 1), etc.
[0035] In this embodiment, since generating pseudo-labels for training images using a professional detector may introduce a large amount of noise, which can negatively impact model training accuracy and lead to accumulated detection errors, it is necessary to filter the generated pseudo-labels after generation. The label quality of each pseudo-label is evaluated based on the label information included in each of the N initial pseudo-label sets. At least some of the pseudo-labels in at least one of the N initial pseudo-label sets are identified as having poor quality, and these poor-quality pseudo-labels are deleted. Poor-quality pseudo-labels may include misidentified pseudo-labels (e.g., misidentifying a triangle as a star), pseudo-labels with inaccurate bounding boxes (e.g., the bounding box does not completely select the object, or the object does not exist within the bounding box), or pseudo-labels with low confidence levels.
[0036] As an optional implementation, based on the label information included in the pseudo-labels of the N initial pseudo-label sets, at least a portion of the pseudo-labels in at least one of the N initial pseudo-label sets are deleted to obtain N target pseudo-label sets. This includes performing the following steps to obtain the i-th target pseudo-label set among the N target pseudo-label sets: determining the quality score of each pseudo-label in the i-th initial pseudo-label set based on the label information included in each pseudo-label in the i-th initial pseudo-label set, and obtaining the i-th set of quality scores; determining whether there are pseudo-labels to be deleted in the i-th initial pseudo-label set based on the i-th set of quality scores and the pre-obtained i-th set of real labels, and deleting the pseudo-labels to be deleted in the i-th initial pseudo-label set if it is determined that there are pseudo-labels to be deleted in the i-th initial pseudo-label set, to obtain the i-th target pseudo-label set. The i-th set of real labels includes real labels obtained by pre-annotating objects in the i-th training image, and each real label in the i-th set of real labels is used to identify an object in the i-th training image.
[0037] The aforementioned ground truth labels can be labels obtained by pre-annotating training images, used to identify at least one object included in the training images, and the category of each object. In the embodiments of this application, ground truth labels can be represented by a binary tuple, such as ( , ),in, This represents the coordinates of the bounding box (i.e., the object detection box, the rectangular region used to select the j-th object) in the training image. The coordinates of the bounding box can be represented by the coordinates of the top-left and bottom-right vertices of the bounding box. This represents the category label of the j-th object in the training image; alternatively, the true label can be represented by a triplet, such as ( , , ),in, This represents the confidence level used to identify the true label of the j-th object in the training image. The default confidence level is 1.
[0038] The aforementioned quality score can be a specific numerical value used to quantify the quality of pseudo-labels. The worse the quality of a pseudo-label, the lower its quality score; conversely, the better the quality, the higher its quality score. In this embodiment, the quality score of a pseudo-label can be determined based on the label information included in the pseudo-label (such as the coordinates of the bounding box, category label, confidence level, etc.).
[0039] In this embodiment, when filtering the pseudo-labels included in the N initial pseudo-label sets, the pseudo-labels included in each of the N initial pseudo-label sets can be filtered sequentially. When filtering the i-th initial pseudo-label set in the N initial pseudo-label sets, the real labels obtained by pre-annotating the i-th training image are first obtained, resulting in the i-th real label set corresponding to the i-th training image. Then, the quality score of each pseudo-label is determined based on the label information included in each pseudo-label in the i-th initial pseudo-label set. For example, if the j-th object identified by the j-th pseudo-label included in the i-th initial pseudo-label set has a real label, the quality score of the j-th pseudo-label can be determined based on the bounding box coordinates included in the j-th pseudo-label. Alternatively, if the j-th object identified by the j-th pseudo-label included in the i-th initial pseudo-label set does not have a real label, the quality score of the j-th pseudo-label can be determined based on the bounding box coordinates included in the j-th pseudo-label. The confidence level and category label are used to determine the quality score of the j-th pseudo-label. Then, based on the quality score of each pseudo-label in the i-th initial pseudo-label set and the i-th real label set, it is determined whether there are pseudo-labels to be deleted in the i-th initial pseudo-label set (such as pseudo-labels with poor label quality). If it is determined that there are pseudo-labels to be deleted in the i-th initial pseudo-label set, the pseudo-labels to be deleted in the i-th initial pseudo-label set are deleted to obtain the i-th target pseudo-label set of the i-th training image. If it is determined that there are no pseudo-labels to be deleted in the i-th initial pseudo-label set, the i-th initial pseudo-label set is directly determined as the i-th target pseudo-label set of the i-th training image.
[0040] Example, Figure 4 This is a schematic diagram of a real label according to an embodiment of this application, such as... Figure 4 As shown, the dashed boxes represent pseudo-labels obtained by detecting and recognizing the training images using a professional detector, while the dotted boxes represent the real labels obtained by manually annotating the training images beforehand. Since manual annotation may only annotate a portion of the objects in the training images, resulting in a small sample size, a professional detector is needed to detect and recognize the training images, output pseudo-labels, and filter these pseudo-labels to obtain higher-quality labels, thus obtaining more labeled samples. For pseudo-labels that have real labels (i.e., the j-th object identified by the j-th pseudo-label has a real label), such as... Figure 3 If pseudo-labels 1-3 correspond to real labels 1-3 respectively (i.e., the pseudo-labels and their corresponding real labels identify the same target), then the quality score of the pseudo-label can be determined based on the bounding box coordinates included in the pseudo-label. For pseudo-labels that do not have real labels (i.e., the j-th object identified by the j-th pseudo-label does not have a real label), such as... Figure 3For pseudo-labels 4-7, their quality scores can be determined based on the category labels and confidence levels they include. Therefore, each pseudo-label is filtered based on its quality score in the training image (i.e., each pseudo-label in the i-th initial pseudo-label set) and the true label in the training image (i.e., the i-th true label set).
[0041] Through the above operations, the quality of pseudo-labels is quantified by the multi-dimensional label information included in the pseudo-labels, and the quality score of each pseudo-label is obtained. Based on the quality score, pseudo-labels are screened, so low-quality pseudo-labels can be dynamically filtered from three aspects: label space matching degree, model cognitive consistency, and data distribution rationality. This achieves a quantitative assessment of the reliability of pseudo-labels, thereby retaining effective supplementary labeled samples while avoiding the introduction of noise.
[0042] Step S206: Train the neural network model to be trained based on N sets of target pseudo-labels and N training images to obtain the target neural network model.
[0043] In this embodiment of the application, after filtering the pseudo-labels in the N initial pseudo-label sets to obtain N target pseudo-label sets, the N target pseudo-label sets and the N training images corresponding to the N target pseudo-label sets are used to train the neural network model to be trained (such as a multimodal model) to obtain the target neural network model.
[0044] Through the above steps, objects in each of the N training images are detected, and pseudo-labels are generated to identify the objects in each training image, resulting in N initial pseudo-label sets for the N training images. Then, based on the label information included in each pseudo-label in the N initial pseudo-label sets, the pseudo-labels included in the N initial pseudo-label sets are filtered, and at least a portion of the pseudo-labels in at least one of the N initial pseudo-label sets are deleted, resulting in N target pseudo-label sets for the N training images. Finally, the neural network model to be trained is trained based on the N target pseudo-label sets and the N training images to obtain the target neural network model.
[0045] By filtering the generated pseudo-labels using multi-dimensional label information included in the pseudo-labels, and dynamically deleting at least a portion of the pseudo-labels from the pseudo-label set, low-quality pseudo-labels are effectively filtered out. This retains valid supplementary samples while avoiding the introduction of noise, reducing the accumulation of model errors caused by noise in training samples during model training, thus improving the model's training accuracy and detection accuracy. Therefore, this approach can solve the technical problem of low detection accuracy in related technologies, achieving the technical effect of improving the model's detection accuracy.
[0046] As an optional implementation, based on the tag information included in each pseudo-tag in the i-th initial pseudo-tag set, the quality score of each pseudo-tag in the i-th initial pseudo-tag set is determined to obtain the i-th set of quality scores, including: performing the following steps to obtain the quality score of the j-th pseudo-tag in the i-th initial pseudo-tag set, wherein the i-th set of quality scores includes the quality score of the j-th pseudo-tag, and j is an integer greater than 0: verifying the j-th pseudo-tag based on the tag information of the j-th pseudo-tag to determine whether the j-th pseudo-tag passes the verification; if the j-th pseudo-tag passes the verification, determining the quality score of the j-th pseudo-tag based on the confidence level included in the j-th pseudo-tag, wherein the confidence level included in the j-th pseudo-tag is used to identify the probability that the category of the j-th object is correct and / or the probability that the bounding box coordinates included in the j-th pseudo-tag are correct; if the j-th pseudo-tag fails the verification, a pre-determined first value is determined as the quality score of the j-th pseudo-tag.
[0047] In this embodiment, when determining the quality score of each pseudo-label, each pseudo-label first needs to be verified to determine its validity, such as whether the position of the bounding box corresponding to the pseudo-label is correct, or whether the category label of the object selected by the identified bounding box is correct, etc.; then, when the pseudo-label is determined to be valid (i.e., the verification passes), the accuracy of the pseudo-label is further determined, such as whether the size of the bounding box is appropriate when selecting objects (i.e., whether it can select the complete object), or whether the confidence of the category label is large enough, etc., thereby quantifying the actual situation of the pseudo-label (described by multi-dimensional label information) into the quality score of the pseudo-label to describe the accuracy of the pseudo-label; and for invalid pseudo-labels (i.e., the verification fails), their quality score is directly determined to a preset first value (such as 0), to indicate that the label quality of the pseudo-label is poor.
[0048] By taking into account the label space matching degree, model cognitive consistency, and data distribution rationality, the quality score of each pseudo-label is dynamically determined, which improves the accuracy of pseudo-label screening, helps to distinguish low-quality pseudo-labels, and thus improves the quality of labeled samples.
[0049] As an optional implementation, based on the quality score of the i-th group and the pre-obtained i-th real tag set, it is determined whether there are pseudo tags to be deleted in the i-th initial pseudo tag set, including at least one of the following: If there is no quality score less than a preset score threshold in the i-th group of quality scores, and the number of pseudo tags included in the i-th initial pseudo tag set is less than or equal to M, it is determined that there are no pseudo tags to be deleted in the i-th initial pseudo tag set, where M represents the number of real tags included in the i-th real tag set, or M is a value obtained by multiplying the number of real tags included in the i-th real tag set by a preset coefficient; if there is a quality score less than a preset score threshold in the i-th group of quality scores, it is determined that there are no pseudo tags to be deleted in the i-th initial pseudo tag set. Given a quality score with a preset score threshold, it is determined that there are pseudo-labels to be deleted in the i-th initial pseudo-label set, and pseudo-labels with quality scores lower than the preset score threshold in the i-th initial pseudo-label set are identified as pseudo-labels to be deleted; if there are no quality scores lower than the preset score threshold in the i-th group of quality scores, but the number of pseudo-labels included in the i-th initial pseudo-label set is greater than M, it is determined that there are pseudo-labels to be deleted in the i-th initial pseudo-label set, and R pseudo-labels in the i-th initial pseudo-label set are identified as pseudo-labels to be deleted, where R pseudo-labels include all pseudo-labels in the i-th initial pseudo-label set except for the M pseudo-labels with the highest quality scores, and R is an integer greater than 0.
[0050] The aforementioned M can be a pre-set threshold for filtering false labels, used to balance the number of real and false labels in a single training image, ensuring that false labels do not dominate the training process and maintaining a balanced data distribution. In this embodiment, M can be directly taken as the number of real labels included in the set of real labels of the training image, or it can be taken as the value obtained by multiplying the number of real labels included in the set of real labels of the training image by a preset coefficient γ.
[0051] In the embodiments of this application, the appendix is used. Figure 4 Taking the pseudo-labels and real labels shown as an example, this training image has 3 real labels and 7 pseudo labels. The specific logic for determining the pseudo labels to be deleted is as follows:
[0052] When M is directly set to the number of real labels in the training image's real label set, which is 3, if the quality scores of pseudo-labels 1-7 are all greater than or equal to the preset score threshold, then the 3 pseudo-labels with the highest quality scores are retained, and the 4 pseudo-labels with the lowest quality scores are designated as pseudo-labels to be deleted and deleted. If the quality scores of pseudo-labels 1 and 7 are less than the preset score threshold, then pseudo-labels 1 and 7 are designated as pseudo-labels to be deleted and deleted. Then, if the number of the remaining 5 pseudo-labels is greater than 3, then the 2 pseudo-labels with the lowest quality scores among the remaining 5 pseudo-labels are designated as pseudo-labels to be deleted and deleted, while the 3 pseudo-labels with the highest quality scores are retained. If the quality scores of pseudo-labels 1 to 5 are less than the preset score threshold, then pseudo-labels 1 to 5 are designated as pseudo-labels to be deleted and deleted. Then, if the number of the remaining 2 pseudo-labels is less than 3, then the remaining pseudo-labels 6 and 7 are directly retained without further deletion.
[0053] When M is the value obtained by multiplying the number of real labels included in the set of real labels of the training image by the preset coefficient γ=2.5, M is 7.5. If the quality scores of pseudo labels 1-7 are all greater than or equal to the preset score threshold, it is confirmed that there are no pseudo labels to be deleted and all pseudo labels are retained. If the quality scores of pseudo labels 1 and pseudo labels 7 are less than the preset score threshold, pseudo labels 1 and pseudo labels 7 are first identified as pseudo labels to be deleted and deleted, and the remaining pseudo labels are retained.
[0054] Through the above operations, by introducing the relevant parameter M, dynamic control of the quality and reasonable limitation of the number of pseudo-labels are achieved. Thus, in the absence of large-scale high-quality labeled data, pseudo-labels can be automatically screened and low-quality pseudo-labels can be eliminated, effectively reducing the dependence of multimodal detection on large-scale high-quality labeled data.
[0055] As an optional implementation, the j-th pseudo-label is validated based on its label information to determine whether it passes validation. This includes: if the j-th object identified by the j-th pseudo-label has a corresponding real label in the i-th real label set, determining the intersection-union ratio (IU / R) between the first bounding box and the second bounding box, where the first bounding box is the bounding box identified by the bounding box coordinates included in the j-th pseudo-label, the bounding box coordinates included in the j-th pseudo-label are used to identify the position of the j-th object in the i-th training image, the label information includes the bounding box coordinates, and the second bounding box is the bounding box identified by the bounding box coordinates included in the corresponding real label; if the IU / R is greater than or equal to a preset first validation threshold, the j-th pseudo-label is determined to pass validation; if the IU / R is less than the first validation threshold, the j-th pseudo-label is determined to fail validation.
[0056] In this application embodiment, for pseudo-labels that have real labels, as shown in the attached... Figure 4 The pseudo-labels 1-3 can be validated based on the position of their corresponding real labels' bounding boxes to determine if the pseudo-labels' bounding boxes are correctly positioned. (See attached image.) Figure 4 Taking pseudo-label 1 as an example, the specific steps are as follows: Calculate the intersection-union ratio (IUGR) between the bounding box (i.e., the dashed box) of pseudo-label 1 and the bounding box (i.e., the dotted box) of real label 1. Since the position of the bounding box of real label 1 is accurate, the IUGR between the two bounding boxes is used. If the IUGR is greater than or equal to the preset first verification threshold, it means that the bounding box of pseudo-label 1 overlaps sufficiently with the bounding box of real label 1, and the position of the bounding box of pseudo-label 1 is considered correct. At this time, pseudo-label 1 is determined to pass the verification (i.e., it is valid); otherwise, pseudo-label 1 is determined to fail the verification (i.e., it is invalid).
[0057] For example, the intersection-union ratio between the first and second bounding boxes can be determined using the following formula:
[0058]
[0059] in, It is the intersection-union ratio between the first bounding box and the second bounding box. It is the bounding box coordinates included by the j-th pseudo-label, and the bounding box coordinates are in the form of , These are the coordinates of the top-left vertex of the bounding box. This represents the coordinates of the bottom right corner vertex of the bounding box. It is the bounding box coordinates of the real label corresponding to the j-th pseudo-label, and the bounding box coordinates are in the form of , These are the coordinates of the top-left vertex of the bounding box. It represents the coordinates of the lower right corner vertex of the bounding box.
[0060] For example, if the j-th object identified by the j-th pseudo-label has a corresponding real label in the i-th real label set, the quality score of the j-th pseudo-label is determined based on the confidence level included in the j-th pseudo-label, including: determining the sum of the intersection-union ratio and a predetermined second value as a first sum; determining the product of the first sum and the confidence level included in the j-th pseudo-label as a first product; and determining the ratio of the first product to a predetermined third value as the quality score of the j-th pseudo-label.
[0061] In this embodiment, the quality score of the j-th pseudo-label can be determined using the following formula:
[0062]
[0063] in, It is the quality score of the j-th pseudo-label. It is the confidence level included in the j-th pseudo-label. It is the intersection-union ratio (IoU) between the first bounding box (i.e., the bounding box identified by the coordinates of the bounding box included in the j-th pseudo-label) and the second bounding box (i.e., the bounding box identified by the coordinates of the bounding box included in the real label corresponding to the j-th pseudo-label). It is a predetermined second value. It is a predetermined third value.
[0064] As an optional implementation, the j-th pseudo-label is validated based on its label information to determine whether it passes validation. This includes: if the j-th object identified by the j-th pseudo-label does not have a corresponding real label in the i-th real label set, obtaining the s-th auxiliary label set. The s-th auxiliary label set includes auxiliary labels obtained by recognizing objects in the i-th training image belonging to the category labels included in the j-th pseudo-label. The category labels included in the j-th pseudo-label are used to identify the category of the j-th object. The label information includes the category label. Each auxiliary label in the s-th auxiliary label set is used for... Identify an object in the i-th training image; if the j-th object identified by the j-th pseudo-label has a corresponding auxiliary label in the s-th auxiliary label set, and the confidence level of the corresponding auxiliary label is greater than or equal to the preset second verification threshold, then the j-th pseudo-label is determined to have passed verification; if the j-th object identified by the j-th pseudo-label does not have a corresponding auxiliary label in the s-th auxiliary label set, or if the j-th object identified by the j-th pseudo-label has a corresponding auxiliary label in the s-th auxiliary label set, but the confidence level of the corresponding auxiliary label is less than the second verification threshold, then the j-th pseudo-label is determined to have failed verification.
[0065] The aforementioned auxiliary label can be a label output by a lightweight multimodal model after detecting and recognizing objects of a certain category in the training image. It is used to identify at least one object under a category label included in the training image. Essentially, this auxiliary label is also a pseudo-label. In this embodiment, the auxiliary label can also be represented by a triplet data, such as ( , , ),in, This represents the coordinates of the bounding box (i.e., the object detection box, the rectangular region used to select the j-th object) in the training image. The coordinates of the bounding box can be represented by the coordinates of the top-left and bottom-right vertices of the bounding box. This represents the category label of the j-th object in the training image. This represents the confidence level of the output of the lightweight multimodal model, which is also the confidence level of the auxiliary label used to identify the j-th object in the training image. The confidence level ranges from [0,1].
[0066] In this application embodiment, for pseudo-labels containing false labels, as shown in the attached... Figure 4 For pseudo-labels 4-7, since there is no accurate real label for comparison, a new pseudo-label needs to be generated as a control group during the validation process to verify whether the category label of the object selected by the pseudo-label's bounding box is correct. (See attached...) Figure 4 Taking pseudo-label 4 as an example, the process is as follows: If pseudo-label 4 identifies triangle 1, and the label category "triangle" included in pseudo-label 4 is used as an index, the detection instruction "detect all triangles in the image" is executed to obtain the corresponding auxiliary label set. If, through the auxiliary label set, there exists an auxiliary label that identifies triangle 1, and the confidence level of that auxiliary label is greater than or equal to the preset second verification threshold, then it is considered that the object identified by pseudo-label 4 is likely a triangle, and the category label of the object selected by the bounding box of pseudo-label 4 is correct. In this case, pseudo-label 4 is deemed to have passed verification (i.e., it is valid). If, in the auxiliary label set, there is no auxiliary label that identifies triangle 1, or if there is an auxiliary label that identifies triangle 1, but the confidence level of that auxiliary label is less than the preset second verification threshold, then it is considered that the object represented by label 4 is highly likely to be a non-triangle, and in this case, pseudo-label 4 is deemed to have failed verification (i.e., it is invalid).
[0067] For example, when the j-th object identified by the j-th pseudo-label does not have a corresponding real label in the i-th real label set, the quality score of the j-th pseudo-label is determined based on the confidence level included in the j-th pseudo-label, including: determining the average of the confidence level included in the j-th pseudo-label and the confidence level included in the corresponding auxiliary label as a first average value; and determining the product of the first average value and a predetermined fourth value as the quality score of the j-th pseudo-label.
[0068] In this embodiment, the quality score of the j-th pseudo-label can be determined using the following formula:
[0069]
[0070] in, It is the quality score of the j-th pseudo-label. It is the confidence level included in the j-th pseudo-label. That is, the confidence level of the auxiliary labels corresponding to the j-th pseudo-label. It is the fourth predetermined value.
[0071] As an optional implementation, training a neural network model to be trained based on N target pseudo-label sets and N training images to obtain a target neural network model includes: obtaining N first training label sets, wherein the i-th first training label set among the N first training label sets is obtained through the following steps: fusing the i-th target pseudo-label set among the N target pseudo-label sets with a pre-obtained i-th real label set, wherein the i-th real label set includes real labels obtained by pre-annotating objects in the i-th training image, and each real label in the i-th real label set is used to identify an object in the i-th training image. Each first training label in the N training images is used to identify an object in the i-th training image. Based on the N first training label sets, N first text data sets of N training images are generated. The i-th first text data set in the N first text data sets includes at least one first text data pair. Each first text data pair in the i-th first text data set is used to identify at least one object in the i-th training image. Each first text data pair includes first positive text data and first negative text data corresponding to the first positive text data. The target neural network model is obtained by training the neural network model to be trained based on the N first training label sets, the N training images, and the N first text data sets.
[0072] The aforementioned first training label can be the label used during the actual training of the neural network model to be trained. Each first training label is used to identify an object in a training image, and this label can be obtained through the aforementioned pseudo-labels and / or real labels. In the embodiments of this application, the training label can be represented by a triplet data, such as ( , , ),in, This represents the coordinates of the bounding box (i.e., the object detection box, the rectangular region used to select the j-th object) ultimately used in the training image. The coordinates of the bounding box can be represented by the coordinates of the top-left and bottom-right vertices of the bounding box. This represents the category label of the j-th object in the training image. This represents the confidence level of the training label used to identify the j-th object in the training image, with the confidence level ranging from [0,1].
[0073] In this embodiment, the i-th target pseudo-label set and the i-th real label set are fused to obtain the i-th first training label set. This can be achieved by merging the i-th target pseudo-label set and the i-th real label set, and then using the NMS (Non-Maximum Suppression) algorithm to remove redundancy from the merged set, resulting in the i-th first training label set for the i-th training image. In this case, the pseudo-labels in the i-th target pseudo-label set or the real labels in the i-th real label set are directly used as the first training labels in the i-th first training label set. The quality score of each first training label also follows the quality score of the pseudo-label or real label, with the quality score of the real label defaulting to 1. For example, see the attached diagram. Figure 4 As shown in the figure, for the training image, if the target pseudo-label set includes pseudo-labels 1-7 and the real label set includes real labels 1-3, the NMS algorithm is used to remove redundancy from the 10 labels in the training image, retaining only the best label for each object in the training image as the final training label for that object, thus obtaining the first training label set of the training image. At this time, the first training label set includes 7 training labels (i.e., the first training labels), namely {training label 1, training label 2, ..., training label 7}. The quality score of each training label is the quality score of the only best label (i.e., pseudo label or real label) for the object it identifies. In this process, the confidence level of the real label is 1 by default, and the quality score of the real label is 1 by default.
[0074] In this embodiment, the i-th target pseudo-label set is fused with the i-th real label set to obtain the i-th first training label set. Alternatively, each pseudo-label in the i-th target pseudo-label set can be weighted and averaged with its corresponding real label in the i-th real label set, and this average can be used as a first training label in the i-th first training label set, thus obtaining the i-th first training label set for the i-th training image. At this time, the bounding box coordinates included in the training label of the j-th object in the i-th training image are... ) is the weighted average of the bounding box coordinates of the pseudo-label and the true label of the j-th object, where the training labels of the j-th object include the category labels ( ) represents the category label of the pseudo-label of the j-th object, and the confidence level () is included in the training label of the j-th object. The confidence score of the pseudo-label of the j-th object is the weighted average of the confidence scores of the real label of the j-th object; simultaneously, the quality score of the training label of the j-th object is also taken as the weighted average of the quality scores of the pseudo-label and the real label of the j-th object. In this process, the confidence score of the real label is 1 by default, and the quality score of the real label is 1 by default.
[0075] The aforementioned first text data pair can be text data generated based on training labels in training images, wherein each first text data pair includes a first positive text data and a first negative text data. In the embodiments of this application, both the first positive text data and the first negative text data are represented in the form of a triple [instruction text, instruction label, image]. The instruction text is generated through a unified instruction template. For the first positive text data, the default instruction label is a match (i.e., positive), and for the first positive text data, the default instruction label is a mismatch (i.e., negative).
[0076] For example, generating N first text data sets of N training images based on N first training label sets includes: performing the following steps to obtain the i-th first text data set among the N first text data sets: filling a first category label into a predetermined text template to generate first text, wherein the first category label is a category label included in any first training label in the i-th first training label set; determining the first text, the predetermined first label, and the i-th training image as the first target positive text data of the i-th training image; filling a second category label into a predetermined text template to generate second text, wherein the second category label is any category other than the category labels included in the first training labels in the i-th first training label set; determining the second text, the predetermined second label, and the i-th training image as the first target negative text data corresponding to the first target positive text data; and determining the first target positive text data and the first target negative text data as the first text data pair in the i-th first text data set.
[0077] Figure 5 This is a schematic diagram of training labels according to an embodiment of this application, such as... Figure 5 As shown, the training image includes 7 objects and 7 training labels for each object. The category label of any training label is selected as the category label (i.e., the first target text data) for generating the training image. For example, the category label "star" from training label 5 is used as the category label for generating the first text data (i.e., the first target text data). The first text (i.e., the instruction text) "Detect all stars in the image" is generated using the instruction template (i.e., the pre-determined text template). The first text, the first label (i.e., the instruction label), "match," and the training image are then combined to generate a triplet as the first target text data. ['Detect all stars in the image', match, training image]; Simultaneously, it is determined that the category labels included in the matched image are triangle, circle, and star. Then, a category label not present in the training image from the pre-determined categories, such as "square," is used as the category label for generating the first negative text data (i.e., the first target negative text data) of the training image. A second text (i.e., the instruction text) "Detect all squares in the image" is generated using the instruction template (i.e., the pre-determined text template). The instruction text, the second label (i.e., the instruction label) "do not match," and the training image are combined to form a triplet as the first target negative text data ['Detect all squares in the image', do not match, training image]. This generates a first text data pair for the training image, ensuring that each training image corresponds to a set of positive and negative instruction samples, thus making the number of image samples consistent with the number of text data, avoiding training data imbalance during model training.
[0078] Through the above operations, after determining the training label set of training images, text data for each training image is generated based on the obtained training label set. Thus, the neural network model to be trained is trained by combining the training images, training label set, and text data, supervising the model to learn the correspondence between instruction text and visual targets, reducing the false detection rate of non-target categories, and improving the detection accuracy of the model.
[0079] As an optional implementation, training a neural network model to be trained based on N first training label sets, N training images, and N first text data sets to obtain a target neural network model includes: performing a first iteration of training on the neural network model to be trained based on the N first training label sets, N training images, and first text data sets until a preset trigger condition is reached, at which point the first iteration of training is stopped, resulting in an intermediate neural network model; generating N second training label sets and N second text data sets based on preset third and fourth verification thresholds, wherein each second training label in the N second training label sets is used to identify an object in a training image, and the N second text data sets include multiple second text data pairs, each second text data pair being used to identify at least one object in a training image; each second text data pair includes second positive text data and second negative text data corresponding to the second positive text data; performing a second iteration of training on the intermediate neural network model based on the N second training label sets, N training images, and N second text data sets until the loss value of the model obtained after the second iteration of training is less than a preset loss threshold, at which point the second iteration of training is stopped, resulting in the target neural network model.
[0080] The aforementioned pre-set trigger conditions can serve as the basis for stopping training during the first iteration of training. This basis can be dynamically adjusted based on the actual training status of the model and its actual needs. Optionally, the pre-set trigger conditions include at least one of the following: the first iteration of training reaches a preset number of training iterations; during the first iteration of training, the model accuracy improvement after a preset number of consecutive iterations is less than a preset accuracy value compared to the model obtained after the previous iteration.
[0081] In the embodiments of this application, when training the neural network model to be trained, different training data can be used in different training rounds to increase the training effect of the model. For example, the trigger conditions are set as periodic trigger (i.e., the model training cycle reaches a cumulative total of 6 cycles) and performance trigger (i.e., during model training, the average accuracy of the validation set increases by less than 0.2 percentage points for two consecutive cycles). When training the neural network model to be trained, the model is first trained using N sets of first training labels, N training images, and a first set of text data. The first iteration training is stopped when the model meets at least one of the above trigger conditions, resulting in an intermediate neural network model. Then, the preset first verification threshold is replaced with a preset third verification threshold, and the preset second verification threshold is replaced with a preset fourth verification threshold. The pseudo-labels in the N initial pseudo-label sets are then re-selected in the same way. Based on the pseudo-label sets obtained after the re-selection, N sets of second training labels and N sets of second text data are generated in the same way. The model is then trained again using the N sets of second training labels, N training images, and N sets of second text data (i.e., the intermediate neural network model is trained a second time). When the loss value of the neural network model obtained after training is less than a preset loss threshold, the training of the model is stopped, resulting in the target neural network model.
[0082] By combining the above operations with dynamic adjustment of the selection strategy based on model training phase and performance changes, and adapting to the evolution of model capabilities, training samples can be generated through relatively lenient selection criteria in the early stages of model training to retain more potentially effective samples and increase the training speed of the model. In the middle and later stages of model training, fewer training samples can be generated through stricter selection criteria to filter out low-quality noise samples and improve the detection accuracy of the model. Thus, by dynamically adjusting the model training samples, the data quality and model capabilities are ensured to iterate synchronously, avoiding model overfitting and capability imbalance.
[0083] As an optional implementation, the loss value of the model obtained after the k-th iteration training in the second iteration training process is determined as follows, where k is an integer greater than 0: the detection loss of the model obtained after the k-th iteration training is determined based on N sets of second training labels, wherein the detection loss is used to optimize the object detection accuracy of the intermediate neural network model; the language modeling loss of the model obtained after the k-th iteration training is determined based on the second text data set, wherein the language modeling loss is used to maintain the cross-modal language understanding ability of the intermediate neural network model; and the weighted sum of the detection loss and the language modeling loss is determined as the loss value of the model obtained after the k-th iteration training.
[0084] For example, the loss value of the model obtained after the kth iteration of training can be determined by the following formula:
[0085]
[0086] In the formula, It is the loss value of the model obtained after the k-th iteration of training. It is the detection loss of the model obtained after the k-th iteration of training. It is the language modeling loss of the model obtained after the k-th iteration of training. and These are predetermined weight values.
[0087] Through the above operations, by constructing a weighted sum of detection loss and language modeling loss as a collaborative optimization objective, the model is supervised to learn the correspondence between instruction text and visual targets. This enables the target neural network model to effectively maintain the model's cross-modal language understanding ability while optimizing the model's target detection accuracy, thereby improving the overall performance of the target neural network model in multi-task and multi-modal scenarios and enhancing the model's generalization ability.
[0088] Optionally, determining the detection loss of the model obtained after the kth round of iteration training based on N sets of second training labels includes: obtaining the detection loss of each second training label included in the N sets of second training labels, wherein the detection loss of the tth second training label in the N sets of second training labels is obtained through the following steps: obtaining the bounding box coordinates, class label, and confidence score of the tth second training label included in the N sets of second training labels, as well as the quality score of the tth second training label, wherein the tth second training label is used to identify the jth object in the i-th training image, and j and t are integers greater than 0; determining the detection loss of the tth second training label based on the bounding box coordinates, class label, and confidence score of the tth second training label, as well as the quality score of the tth second training label; and determining the sum of the detection losses of each second training label included in the N sets of second training labels as the detection loss of the model obtained after the kth round of iteration training.
[0089] As an optional implementation, the detection loss of the t-th second training label is determined based on the bounding box coordinates, class label, and confidence score of the t-th second training label, and the quality score of the t-th second training label. This includes: determining a bounding box regression loss based on the bounding box coordinates of the t-th second training label and the bounding box coordinates output after the k-th iteration training, wherein the bounding box regression loss is used to represent the localization error between the bounding box of the t-th second training label and the bounding box output after the k-th iteration training; determining a classification loss based on the class label of the t-th second training label and the class label output after the k-th iteration training, wherein the classification loss is used to represent the difference between the class label of the t-th second training label and the class label output after the k-th iteration training; determining the sum of the bounding box regression loss and the classification loss as a second sum; and determining the detection loss of the t-th second training label as the product of the quality score of the t-th second training label, the confidence score of the t-th second training label, and the second sum.
[0090] For example, the detection loss of the model obtained after the kth iteration of training can be determined by the following formula:
[0091]
[0092] In the formula, It is the detection loss of the model obtained after the k-th iteration of training. , , It is a training label (such as the t-th training label) from a set of N second training labels. After the k-th iteration of training, the model will output a detection label corresponding to each object identified by the second training label. The detection label output by the model after the k-th iteration of training and the corresponding second training label are used to identify the same object. This is the summation symbol, used to sum the detection losses of each of the N second training labels included in the second training label set. The detection loss of each second training label is also the loss between each second training label and the model output after the k-th iteration of training, and its corresponding label. This is the quality score of the t-th training label. This quality score is determined based on the quality scores of the pseudo-labels and / or the real labels calculated above. The quality score of the real labels is set to 1 by default. It represents the confidence level of the t-th training label. The model, obtained after the k-th iteration of training, outputs the bounding box coordinates of the detection label for the t-th object identified by the second training label. It represents the bounding box coordinates included in the t-th training label. It is the localization error between the bounding box output by the model after the k-th iteration of training for the object identified by the t-th second training label and the bounding box indicated by the t-th training label. The model obtained after the k-th iteration of training outputs a detection label containing the category label for the object identified by the t-th second training label. It is the category label included in the t-th training label. It is the difference between the class label output by the model after the k-th iteration of training for the object identified by the t-th second training label and the class label indicated by the t-th training label.
[0093] Alternatively, the language modeling loss of the model obtained after the k-th iteration of training can be determined by the following formula:
[0094]
[0095] In the formula, It is the language modeling loss of the model obtained after the k-th iteration of training. This is the summation notation, used to sum the language modeling loss for each of the N training images. It is the image feature of one training image (such as the i-th training image) out of N training images. It is the v-th word element in the second text data pair corresponding to the i-th training image. These are the word units generated by the model before time step v, obtained after the k-th iteration of training. It is the probability that the model obtained after the k-th iteration of training will generate the v-th word at time step v, based on the i-th training image and the words already generated before time step v.
[0096] As an optional implementation, Figure 6 This is a flowchart of model training according to an embodiment of this application, such as... Figure 6 As shown, the model to be trained (i.e., the target neural network model) mainly includes a visual encoder to ensure the integrity of visual features, a projector (such as a linear projector) whose input dimension is consistent with the output of the visual encoder and whose output dimension matches the embedding dimension of the large language model to achieve cross-modal feature alignment, a large language model, and a lightweight validation model (such as a lightweight multimodal model) to generate auxiliary labels to verify pseudo-labels; the specific model training process is as follows:
[0097] At the beginning of model training, the model is controlled to undergo multiple rounds of dynamic iterative training, and the triggering conditions for dynamic updates are set.
[0098] During the first stage of model training (such as the first two iterations), the input consists of a dataset containing multiple sets of images and their corresponding text descriptions. This dataset is used to train only the cross-modal projector, achieving basic alignment between visual and linguistic features. This stage does not trigger a dynamic update mechanism; it merely lays the groundwork for modal adaptation capabilities in subsequent detection training.
[0099] During the second stage of training (i.e., the first iteration training) of the model, the N initial pseudo-label sets generated are filtered using the preset first verification threshold and the preset second verification threshold, resulting in N first training label sets and N first text data sets. Then, the model is trained in the first iteration using the N first training label sets, N first text data sets, and N training images. At the same time, the projector and the output layer of the large language model are trained to help the model quickly learn the distribution of detection features.
[0100] In the first iteration of training, after each round of training, a check is performed to determine whether the current model meets the triggering conditions. If the triggering conditions are not met, the first iteration of training continues. If the triggering conditions are met, the first iteration of training stops, and an intermediate model (i.e., an intermediate neural network model) is obtained. The preset first and second verification thresholds are replaced with preset third and fourth verification thresholds, respectively. Then, the N initial pseudo-label sets generated are filtered in the same way to generate N second training label sets and N second text data sets. The training set is reconstructed, and the third stage of training continues.
[0101] During the third stage of training (i.e., the second iteration training), the model is trained in the second iteration using N second training label sets, N second text data sets, and N training images (i.e., the reconstructed training set). The projector and the output layer of the large language model are trained simultaneously to avoid interference from low-quality pseudo-labels and improve the detection accuracy of the model. The restored training (i.e., the second iteration training) uses the original hyperparameter configuration to form an iterative closed loop between data and model, achieving synchronous execution and dynamic updates.
[0102] In the second iteration of training, after each round of training, a judgment is made to determine whether the loss value of the current model (i.e., the neural network model obtained after the kth round of training) is less than the preset loss threshold. If the loss value of the current model is less than the preset loss threshold, the fourth stage of training continues, or training is stopped directly to obtain the target model (i.e., the target neural network model); if the loss value of the current model is greater than or equal to the preset loss threshold, the second iteration of training continues.
[0103] Optionally, after the detection performance stabilizes, if it is necessary to retain capabilities such as image description, the model can continue to undergo a fourth stage of training (i.e., multimodal supplementary training) to fine-tune the projector and the output layer of the large language model. In this stage, the model uses the existing hyperparameters and does not trigger a dynamic update mechanism, but only adapts to the requirements of multimodal tasks.
[0104] During the fourth stage of model training (such as the last two iterations), after generating a reconstructed instruction dataset based on N sets of second training labels, the model is trained using this reconstructed instruction dataset, and language modeling loss is applied. The output layer of the supervised large language model is fine-tuned to transfer the fine-grained target knowledge (such as category and location) learned in the detection phase to the multimodal dialogue task, enabling the model to have image description capabilities without reducing detection accuracy. After that, training is stopped, and the target model (i.e., the target neural network model) is obtained. The recombined instruction dataset includes multiple recombined instruction samples. Each recombined instruction sample can be represented by a triplet of [question instruction, reference answer, image]. The question instruction and reference answer can be generated by a pre-set question template. The information filled in the template comes from the information indicated by each of the N second training label sets, such as bounding box coordinates and category labels.
[0105] It should be noted that the recombined instruction dataset used in the fourth stage of training differs from the first text dataset used in the second stage and the second text dataset used in the third stage in that the first and second text datasets focus on improving the model's understanding and execution of the specific instruction of "detecting objects of a specified category"; while the recombined instruction dataset is dedicated to generalizing detection knowledge to more open multimodal interactive tasks (such as question answering and description), thereby achieving synergistic supplementation of multimodal capabilities on the "detection-oriented" model.
[0106] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods according to the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method.
[0107] Embodiments of this application also provide a model training apparatus. Figure 7 This is a structural block diagram of a model training apparatus according to an embodiment of this application, such as... Figure 7As shown, the device includes: an acquisition module 702, used to acquire initial pseudo-label sets of each training image in N training images to obtain N initial pseudo-label sets, wherein the i-th initial pseudo-label set includes pseudo-labels obtained by detecting objects in the i-th training image, each pseudo-label in the i-th initial pseudo-label set is used to identify an object in the i-th training image, the N initial pseudo-label sets include the i-th initial pseudo-label set, the N training images include the i-th training image, N is an integer greater than 0, and i is an integer greater than 0 and less than or equal to N; a deletion module 704, used to delete at least a portion of the pseudo-labels in at least one of the N initial pseudo-label sets based on the label information included in the pseudo-labels in the N initial pseudo-label sets to obtain N target pseudo-label sets; and a training module 706, used to train a neural network model to be trained based on the N target pseudo-label sets and the N training images to obtain a target neural network model.
[0108] In an exemplary embodiment, the above-described apparatus is further configured to perform the following steps to obtain the i-th target pseudo-label set among N target pseudo-label sets: based on the label information included in each pseudo-label in the i-th initial pseudo-label set, determine the quality score of each pseudo-label in the i-th initial pseudo-label set to obtain the i-th set of quality scores; based on the i-th set of quality scores and the pre-obtained i-th set of real labels, determine whether there are pseudo-labels to be deleted in the i-th initial pseudo-label set, and if it is determined that there are pseudo-labels to be deleted in the i-th initial pseudo-label set, delete the pseudo-labels to be deleted in the i-th initial pseudo-label set to obtain the i-th target pseudo-label set, wherein the i-th set of real labels includes real labels obtained by pre-annotating objects in the i-th training image, and each real label in the i-th set of real labels is used to identify an object in the i-th training image.
[0109] In an exemplary embodiment, the above-described apparatus is further configured to perform the following steps to obtain the quality score of the j-th pseudo-label in the i-th initial pseudo-label set, wherein the i-th set of quality scores includes the quality score of the j-th pseudo-label, and j is an integer greater than 0: verifying the j-th pseudo-label based on the label information of the j-th pseudo-label to determine whether the j-th pseudo-label passes the verification; if the j-th pseudo-label passes the verification, determining the quality score of the j-th pseudo-label based on the confidence level included in the j-th pseudo-label, wherein the confidence level included in the j-th pseudo-label is used to identify the probability that the category of the j-th object is correct and / or the probability that the bounding box coordinates included in the j-th pseudo-label are correct; if the j-th pseudo-label fails the verification, determining a predetermined first value as the quality score of the j-th pseudo-label.
[0110] In an exemplary embodiment, the apparatus is further configured to, when the j-th object identified by the j-th pseudo-label has a corresponding real label in the i-th real label set, determine the intersection-union ratio (IU / R) between the first bounding box and the second bounding box, wherein the first bounding box is the bounding box identified by the bounding box coordinates included in the j-th pseudo-label, the bounding box coordinates included in the j-th pseudo-label are used to identify the position of the j-th object in the i-th training image, the label information includes the bounding box coordinates, and the second bounding box is the bounding box identified by the bounding box coordinates included in the corresponding real label; if the IU / R is greater than or equal to a preset first verification threshold, determine that the j-th pseudo-label has passed verification; if the IU / R is less than the first verification threshold, determine that the j-th pseudo-label has failed verification.
[0111] In an exemplary embodiment, the above-described apparatus is further configured to, when the j-th object identified by the j-th pseudo-label has a corresponding real label in the i-th real label set, determine the sum of the intersection-union ratio and a predetermined second value as a first sum; determine the product of the first sum and the confidence level included in the j-th pseudo-label as a first product; and determine the ratio of the first product to a predetermined third value as the quality score of the j-th pseudo-label.
[0112] In an exemplary embodiment, the apparatus is further configured to: obtain an s-th auxiliary label set when the j-th object identified by the j-th pseudo-label does not have a corresponding real label in the i-th real label set; wherein the s-th auxiliary label set includes auxiliary labels obtained by identifying objects in the i-th training image that belong to the category labels included in the j-th pseudo-label; the category labels included in the j-th pseudo-label are used to identify the category of the j-th object; the label information includes the category label; each auxiliary label in the s-th auxiliary label set is used to identify an object in the i-th training image; determine that the j-th pseudo-label passes verification when the j-th object identified by the j-th pseudo-label has a corresponding auxiliary label in the s-th auxiliary label set, and the confidence level of the corresponding auxiliary label is greater than or equal to a preset second verification threshold; determine that the j-th pseudo-label fails verification when the j-th object identified by the j-th pseudo-label does not have a corresponding auxiliary label in the s-th auxiliary label set, or when the j-th object identified by the j-th pseudo-label has a corresponding auxiliary label in the s-th auxiliary label set, but the confidence level of the corresponding auxiliary label is less than the second verification threshold.
[0113] In an exemplary embodiment, the above-described apparatus is further configured to, when the j-th object identified by the j-th pseudo-label does not have a corresponding real label in the i-th real label set, determine the average of the confidence level included in the j-th pseudo-label and the confidence level included in the corresponding auxiliary label as a first average value; and determine the product of the first average value and a predetermined fourth value as the quality score of the j-th pseudo-label.
[0114] If there is no quality score below a preset score threshold in the i-th quality score group, and the number of pseudo-labels in the i-th initial pseudo-label set is less than or equal to M, then it is determined that there are no pseudo-labels to be deleted in the i-th initial pseudo-label set, where M represents the number of real labels in the i-th real label set, or M is the value obtained by multiplying the number of real labels in the i-th real label set by a preset coefficient; if there is a quality score below a preset score threshold in the i-th quality score group, then it is determined that there are pseudo-labels to be deleted in the i-th initial pseudo-label set. The system identifies pseudo-labels in the i-th initial pseudo-label set whose quality scores are less than a preset score threshold and determines them as pseudo-labels to be deleted. If there are no quality scores less than the preset score threshold in the i-th group of quality scores, but the number of pseudo-labels included in the i-th initial pseudo-label set is greater than M, then it is determined that there are pseudo-labels to be deleted in the i-th initial pseudo-label set, and R pseudo-labels in the i-th initial pseudo-label set are identified as pseudo-labels to be deleted. Here, R pseudo-labels include all pseudo-labels in the i-th initial pseudo-label set except for the M pseudo-labels with the highest quality scores, and R is an integer greater than 0.
[0115] In an exemplary embodiment, the above-described apparatus is further configured to acquire N first training label sets, wherein the i-th first training label set among the N first training label sets is obtained through the following steps: fusing the i-th target pseudo-label set among the N target pseudo-label sets with a pre-acquired i-th real label set, wherein the i-th real label set includes real labels obtained by pre-annotating objects in the i-th training image, each real label in the i-th real label set is used to identify an object in the i-th training image, and each first training label in the i-th first training label set is used to identify the i-th training image. An object in the first training image; N first text data sets are generated based on N first training label sets, and N training images are generated. The i-th first text data set in the N first text data sets includes at least one first text data pair. Each first text data pair in the i-th first text data set is used to identify at least one object in the i-th training image. Each first text data pair includes first positive text data and first negative text data corresponding to the first positive text data. The target neural network model is obtained by training the neural network model to be trained based on the N first training label sets, N training images and N first text data sets.
[0116] In an exemplary embodiment, the above-described apparatus is further configured to perform the following steps to obtain the i-th first text data set among N first text data sets: filling a first category label into a predetermined text template to generate first text, wherein the first category label is a category label included in any first training label in the i-th first training label set; determining the first text, the predetermined first label, and the i-th training image as the first target positive text data of the i-th training image; filling a second category label into a predetermined text template to generate second text, wherein the second category label is any category other than the category labels included in the first training labels in the i-th first training label set; determining the second text, the predetermined second label, and the i-th training image as the first target negative text data corresponding to the first target positive text data; and determining the first target positive text data and the first target negative text data as the first text data pair in the i-th first text data set.
[0117] In an exemplary embodiment, the apparatus is further configured to perform a first iteration of training on the neural network model to be trained based on N first training label sets, N training images, and a first text data set until a preset trigger condition is met, at which point the first iteration of training is stopped, resulting in an intermediate neural network model; generate N second training label sets and N second text data sets based on preset third and fourth verification thresholds, wherein each second training label in the N second training label sets is used to identify an object in a training image, and the N second text data sets include multiple second text data pairs, each second text data pair being used to identify at least one object in a training image; each second text data pair includes second positive text data and second negative text data corresponding to the second positive text data; and perform a second iteration of training on the intermediate neural network model based on the N second training label sets, N training images, and N second text data sets until the loss value of the model obtained after the second iteration of training is less than a preset loss threshold, at which point the second iteration of training is stopped, resulting in a target neural network model.
[0118] In an exemplary embodiment, the pre-set triggering condition includes at least one of the following: the first iteration training round reaches a preset training round; during the first iteration training, the model obtained after a preset number of consecutive iterations of training has a model accuracy improvement of less than a preset accuracy value compared to the model obtained after the previous iteration training.
[0119] In an exemplary embodiment, the above-described apparatus is further configured to determine the loss value of the model obtained after the k-th iteration training in the second iteration training process by: determining the detection loss of the model obtained after the k-th iteration training based on N second training label sets, wherein the detection loss is used to optimize the target detection accuracy of the intermediate neural network model; determining the language modeling loss of the model obtained after the k-th iteration training based on the second text data set, wherein the language modeling loss is used to maintain the cross-modal language understanding ability of the intermediate neural network model; and determining the loss value of the model obtained after the k-th iteration training by weighted sum of the detection loss and the language modeling loss.
[0120] In an exemplary embodiment, the above-described apparatus is further configured to obtain the detection loss of each second training label included in the N sets of second training labels, wherein the detection loss of the t-th second training label in the N sets of second training labels is obtained by the following steps: obtaining the bounding box coordinates, class label, and confidence score of the t-th second training label included in the N sets of second training labels, as well as the quality score of the t-th second training label, wherein the t-th second training label is used to identify the j-th object in the i-th training image, and j and t are integers greater than 0; determining the detection loss of the t-th second training label based on the bounding box coordinates, class label, and confidence score of the t-th second training label, as well as the quality score of the t-th second training label; and determining the sum of the detection losses of each second training label included in the N sets of second training labels as the detection loss of the model obtained after the k-th iteration of training.
[0121] In an exemplary embodiment, the apparatus is further configured to: determine a bounding box regression loss based on the bounding box coordinates included in the t-th second training label and the bounding box coordinates output after the k-th iteration training, wherein the bounding box regression loss represents the localization error between the bounding box included in the t-th second training label and the bounding box output after the k-th iteration training; determine a classification loss based on the category label included in the t-th second training label and the category label output after the k-th iteration training, wherein the classification loss represents the difference between the category label included in the t-th second training label and the category label output after the k-th iteration training; determine a second sum value by summing the bounding box regression loss and the classification loss; and determine the detection loss of the t-th second training label by multiplying the quality score of the t-th second training label, the confidence level included in the t-th second training label, and the second sum value.
[0122] For a description of the features in the embodiment corresponding to the model training device, please refer to the relevant description of the embodiment corresponding to the model training method, which will not be repeated here.
[0123] Embodiments of this application also provide an electronic device, including a memory and a processor, wherein the memory stores a computer program and the processor is configured to run the computer program to perform the steps in any of the above-described model training method embodiments.
[0124] Embodiments of this application also provide a computer-readable storage medium storing a computer program, wherein the computer program is configured to execute the steps in any of the above-described model training method embodiments at runtime.
[0125] In one exemplary embodiment, the aforementioned computer-readable storage medium may include, but is not limited to, various media capable of storing computer programs, such as a USB flash drive, read-only memory (ROM), random access memory (RAM), portable hard disk, magnetic disk, or optical disk.
[0126] Embodiments of this application also provide a computer program product, which includes a computer program that, when executed by a processor, implements the steps in any of the above-described model training method embodiments.
[0127] Embodiments of this application also provide another computer program product, including a non-volatile computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps in any of the above-described model training method embodiments.
[0128] Any of the components, modules, units, parts, methods, and operations described herein can be implemented using software, firmware, hardware (e.g., fixed logic circuitry), manual processing, or any combination thereof. Alternatively or additionally, any functionality described herein can be executed at least in part by one or more hardware logic components, such as, but not limited to, a central processing unit (CPU), a field-programmable gate array (FPGA), an application-specific integrated circuit (ASIC), an application-specific standard product (ASSP), a system-on-a-chip (SoC), a complex programmable logic device (CPLD), a microprocessor (MCU), etc. The terms "system," "computing device," or "apparatus" as used herein encompass various means, devices, and machines for processing data, including, for example, one or more programmable processors, computers, SoCs, or combinations thereof. The apparatus may also include code that creates an execution environment for the computer program in question, such as code constituting processor firmware, a protocol stack, a database management system, an operating system, a cross-platform runtime environment, a virtual machine, or one or more combinations thereof. The aforementioned computer program (also known as a program, software, software application, app, script, or code) can be written in any form of programming language, including compiled or interpreted languages, declarative or procedural languages, and can be deployed in any form, including as a standalone program or as a module, component, subroutine, object, or other unit suitable for a computing environment.
[0129] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.
[0130] The above provides a detailed description of a model training method provided in this application. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the embodiments above are only intended to help understand the method and its core ideas. It should be noted that those skilled in the art can make various improvements and modifications to this application without departing from its principles, and these improvements and modifications also fall within the protection scope of the claims of this application.
Claims
1. A model training method, characterized in that, include: Obtain the initial pseudo-label set for each of the N training images, resulting in N initial pseudo-label sets. The i-th initial pseudo-label set includes pseudo-labels obtained by detecting objects in the i-th training image. Each pseudo-label in the i-th initial pseudo-label set is used to identify an object in the i-th training image. The N initial pseudo-label sets include the i-th initial pseudo-label set. The N training images include the i-th training image. N is an integer greater than 0, and i is an integer greater than 0 and less than or equal to N. Based on the tag information included in the pseudo-tags in the N initial pseudo-tag sets, at least a portion of the pseudo-tags in at least one of the N initial pseudo-tag sets are deleted to obtain N target pseudo-tag sets. The target neural network model is obtained by training the neural network model to be trained based on the N target pseudo-label sets and the N training images; Specifically, based on the tag information included in the pseudo-tags of the N initial pseudo-tag sets, at least a portion of the pseudo-tags in at least one of the N initial pseudo-tag sets are deleted to obtain N target pseudo-tag sets, including: The following steps are performed to obtain the i-th target pseudo-label set from the N target pseudo-label sets: Based on the tag information included in each pseudo-tag in the i-th initial pseudo-tag set, the quality score of each pseudo-tag in the i-th initial pseudo-tag set is determined to obtain the i-th group of quality scores. Wherein, if the j-th object identified by the j-th pseudo-tag included in the i-th initial pseudo-tag set has a real tag, the quality score of the j-th pseudo-tag is determined based on the bounding box coordinates included in the j-th pseudo-tag. Alternatively, if the j-th object identified by the j-th pseudo-tag included in the i-th initial pseudo-tag set does not have a real tag, the quality score of the j-th pseudo-tag is determined based on the confidence level and category tag included in the j-th pseudo-tag. Here, j is an integer greater than 0. Based on the i-th quality score and the pre-obtained i-th real label set, it is determined whether there are any pseudo-labels to be deleted in the i-th initial pseudo-label set. If it is determined that there are any pseudo-labels to be deleted in the i-th initial pseudo-label set, the pseudo-labels to be deleted in the i-th initial pseudo-label set are deleted to obtain the i-th target pseudo-label set. The i-th real label set includes real labels obtained by pre-annotating objects in the i-th training image. Each real label in the i-th real label set is used to identify an object in the i-th training image. The number of real labels included in the i-th real label set is used to set a number threshold. The number threshold is used to determine whether there are any pseudo-labels to be deleted in the i-th initial pseudo-label set.
2. The method of claim 1, wherein, The quality score of each pseudo-label in the i-th initial pseudo-label set is determined based on the label information included in each pseudo-label, resulting in the i-th set of quality scores, including: The following steps are performed to obtain the quality score of the j-th pseudo-label in the i-th initial pseudo-label set, wherein the i-th set of quality scores includes the quality score of the j-th pseudo-label: The j-th pseudo-tag is verified based on its tag information to determine whether it passes the verification. If the j-th pseudo-label passes the verification, the quality score of the j-th pseudo-label is determined based on the confidence level included in the j-th pseudo-label, wherein the confidence level included in the j-th pseudo-label is used to identify the probability that the category of the j-th object is correct and / or the probability that the bounding box coordinates included in the j-th pseudo-label are correct; If the j-th pseudo-label fails the verification, a predetermined first value will be used as the quality score of the j-th pseudo-label.
3. The method of claim 2, wherein, The step of verifying the j-th pseudo-tag based on its tag information to determine whether the j-th pseudo-tag passes verification includes: If the j-th object identified by the j-th pseudo-label has a corresponding real label in the i-th real label set, the intersection-union ratio (IU / U) between the first bounding box and the second bounding box is determined. The first bounding box is the bounding box identified by the bounding box coordinates included in the j-th pseudo-label. The bounding box coordinates included in the j-th pseudo-label are used to identify the position of the j-th object in the i-th training image. The label information includes the bounding box coordinates. The second bounding box is the bounding box identified by the bounding box coordinates included in the corresponding real label. If the cross-union ratio is greater than or equal to a preset first verification threshold, the j-th pseudo-label is determined to have passed verification. If the cross-union ratio is less than the first verification threshold, it is determined that the j-th pseudo-label verification has failed.
4. The method of claim 3, wherein, Determining the quality score of the j-th pseudo-label based on the confidence level included in the j-th pseudo-label includes: If the j-th object identified by the j-th pseudo-label has a corresponding real label in the i-th real label set, the sum of the intersection-union ratio and the predetermined second value is determined as the first sum value; The product of the first sum and the confidence level included in the j-th pseudo-label is determined as the first product; The ratio of the first product to a predetermined third value is determined as the quality score of the j-th pseudo-label.
5. The method of claim 2, wherein, The step of verifying the j-th pseudo-tag based on its tag information to determine whether the j-th pseudo-tag passes verification includes: If the j-th object identified by the j-th pseudo-label does not have a corresponding real label in the i-th real label set, the s-th auxiliary label set is obtained. The s-th auxiliary label set includes auxiliary labels obtained by recognizing objects in the i-th training image that belong to the category labels included by the j-th pseudo-label. The category labels included by the j-th pseudo-label are used to identify the category of the j-th object. The label information includes the category labels. Each auxiliary label in the s-th auxiliary label set is used to identify an object in the i-th training image. If the j-th object identified by the j-th pseudo-label has a corresponding auxiliary label in the s-th auxiliary label set, and the confidence level of the corresponding auxiliary label is greater than or equal to the preset second verification threshold, then the j-th pseudo-label is determined to have passed the verification. If the j-th object identified by the j-th pseudo-label does not have a corresponding auxiliary label in the s-th auxiliary label set, or if the j-th object identified by the j-th pseudo-label has a corresponding auxiliary label in the s-th auxiliary label set, but the confidence level of the corresponding auxiliary label is less than the second verification threshold, then the verification of the j-th pseudo-label is determined to be unsuccessful.
6. The method of claim 5, wherein, Determining the quality score of the j-th pseudo-label based on the confidence level included in the j-th pseudo-label includes: If the j-th object identified by the j-th pseudo-label does not have a corresponding real label in the i-th real label set, the average of the confidence level included in the j-th pseudo-label and the confidence level included in the corresponding auxiliary label is determined as the first average value; The product of the first average value and the predetermined fourth value is determined as the quality score of the j-th pseudo-label.
7. The method of claim 1, wherein, The step of determining whether there are pseudo-tags to be deleted in the i-th initial pseudo-tag set based on the i-th group quality score and the pre-obtained i-th real tag set includes at least one of the following: If there is no quality score lower than a preset score threshold in the i-th quality score group, and the number of pseudo tags included in the i-th initial pseudo tag set is less than or equal to M, it is determined that there are no pseudo tags to be deleted in the i-th initial pseudo tag set, where M represents the number of real tags included in the i-th real tag set, or M is the value obtained by multiplying the number of real tags included in the i-th real tag set by a preset coefficient; If there is a quality score in the i-th group that is lower than the preset score threshold, it is determined that there is a pseudo-label to be deleted in the i-th initial pseudo-label set, and the pseudo-labels in the i-th initial pseudo-label set whose quality scores are lower than the preset score threshold are determined as the pseudo-labels to be deleted; If there is no quality score lower than the preset score threshold in the i-th quality score group, but the number of pseudo-labels included in the i-th initial pseudo-label set is greater than M, it is determined that there are pseudo-labels to be deleted in the i-th initial pseudo-label set, and R pseudo-labels in the i-th initial pseudo-label set are determined as the pseudo-labels to be deleted, wherein the R pseudo-labels include pseudo-labels in the i-th initial pseudo-label set other than the M pseudo-labels with the highest quality scores, and R is an integer greater than 0.
8. The method according to any one of claims 1 to 7, characterized in that, The process of training the neural network model to be trained based on the N target pseudo-label sets and the N training images to obtain the target neural network model includes: Obtain N sets of first training labels, wherein the i-th set of the N sets of first training labels is obtained through the following steps: merging the i-th set of the N sets of target pseudo-labels with the i-th set of real labels obtained in advance, wherein the i-th set of real labels includes real labels obtained by pre-annotating objects in the i-th training image, each real label in the i-th set of real labels is used to identify an object in the i-th training image, and each first training label in the i-th set of first training labels is used to identify an object in the i-th training image; Based on the N first training label sets, N first text data sets of the N training images are generated, wherein the i-th first text data set of the N first text data sets includes at least one first text data pair, each first text data pair in the i-th first text data set is used to identify at least one object in the i-th training image, and each first text data pair includes first positive text data and first negative text data corresponding to the first positive text data; The target neural network model is obtained by training the neural network model based on the N first training label sets, the N training images, and the N first text data sets.
9. The method of claim 8, wherein, The process of generating N first text data sets from the N training images based on the N first training label sets includes: The following steps are performed to obtain the i-th first text data set from the N first text data sets: The first category label is filled into a predetermined text template to generate the first text, wherein the first category label is the category label included by any first training label in the i-th first training label set; The first text, the pre-determined first label, and the i-th training image are determined as the first target text data of the i-th training image; The second category label is filled into a predetermined text template to generate the second text. The second category label is any category other than the category labels included in the first training labels in the i-th first training label set. The second text, the pre-determined second label, and the i-th training image are determined as the first target negative text data corresponding to the first target positive text data; The first target positive text data and the first target negative text data are determined as the first text data pair in the i-th first text data set.
10. The method of claim 8, wherein, The step of training the neural network model to be trained based on the N first training label sets, the N training images, and the N first text data sets to obtain the target neural network model includes: The neural network model to be trained is subjected to the first iteration of training based on the N first training label sets, the N training images and the first text data set until a preset trigger condition is reached, at which point the first iteration of training is stopped and an intermediate neural network model is obtained. Based on a preset third verification threshold and a preset fourth verification threshold, N second training label sets and N second text data sets are generated. Each second training label in the N second training label sets is used to identify an object in a training image. The N second text data sets include multiple second text data pairs. Each second text data pair in the N second text data sets is used to identify at least one object in a training image. Each second text data pair includes second positive text data and second negative text data corresponding to the second positive text data. The intermediate neural network model is trained in a second iteration based on the N second training label sets, the N training images, and the N second text data sets until the loss value of the model obtained after the second iteration training is less than a preset loss threshold. The second iteration training is then stopped to obtain the target neural network model.
11. The method of claim 10, wherein, The pre-set trigger conditions include at least one of the following: The first iteration of training reached the preset number of training rounds; During the first iteration of training, the model obtained after a preset number of iterations has a lower accuracy improvement than the model obtained after the previous iteration than a preset accuracy value.
12. The method according to claim 10, characterized in that, The loss value of the model obtained after the k-th iteration of the second iteration training process is determined as follows, where k is an integer greater than 0: The detection loss of the model obtained after the kth round of iterative training is determined based on the N second training label sets, wherein the detection loss is used to optimize the target detection accuracy of the intermediate neural network model; The language modeling loss of the model obtained after the kth round of iterative training is determined based on the second text data set, wherein the language modeling loss is used to maintain the cross-modal language understanding ability of the intermediate neural network model; The weighted sum of the detection loss and the language modeling loss is determined as the loss value of the model obtained after the k-th iteration of training.
13. The method of claim 12, wherein, The determination of the detection loss of the model obtained after the k-th iteration of training based on the N second training label sets includes: The detection loss of each second training label included in the N sets of second training labels is obtained, wherein the detection loss of the t-th second training label in the N sets of second training labels is obtained by the following steps: obtaining the bounding box coordinates, class label and confidence of the t-th second training label included in the N sets of second training labels, and the quality score of the t-th second training label, wherein the t-th second training label is used to identify the j-th object in the i-th training image, and j and t are integers greater than 0; The detection loss of the t-th second training label is determined based on the bounding box coordinates, class label, and confidence score of the t-th second training label, as well as the quality score of the t-th second training label. The sum of the detection losses of each of the N second training labels included in the second training label set is determined as the detection loss of the model obtained after the k-th round of iterative training.
14. The method of claim 13, wherein, The step of determining the detection loss for the t-th second training label based on the bounding box coordinates, class label, and confidence score included in the t-th second training label, and the quality score of the t-th second training label, includes: The bounding box regression loss is determined based on the bounding box coordinates included in the t-th second training label and the bounding box coordinates output after the k-th iteration training, wherein the bounding box regression loss is used to represent the localization error between the bounding box included in the t-th second training label and the bounding box output after the k-th iteration training. The classification loss is determined based on the category labels included in the t-th second training label and the category labels output after the k-th iteration training, wherein the classification loss is used to represent the difference between the category labels included in the t-th second training label and the category labels output after the k-th iteration training; The sum of the bounding box regression loss and the classification loss is determined as the second sum. The product of the quality score of the t-th second training label, the confidence level included in the t-th second training label, and the second sum is determined as the detection loss of the t-th second training label.
