Optimization method and device of image description text generation model and computer equipment

Abstract

The invention relates to an optimization method and device for an image description text generation model, computer equipment, a storage medium and a computer program product. The method comprises the following steps: taking an image sample as input data, and obtaining a first word probability set output by an image description text generation model before training and a second word probability set output in a training process; performing open set recognition on the basis of the image sample and an actual description text represented by the second word probability set to obtain a text confidence coefficient of the actual description text; determining model loss based on the second word probability set and the text confidence by taking the first word probability set as a reference; and based on the model loss, adjusting model parameters of the image description text generation model to obtain an optimized image description text generation model. By adopting the method, the accuracy of generating the image description text can be improved.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to an optimization method, apparatus, computer device, and storage medium for an image description text generation model. Background Technology

[0002] With the rapid development of artificial intelligence (AI) technology, AI is being researched and applied in various fields. Image recognition technology enables "image description," which means converting the content information in an image into image descriptive information. Currently, image descriptive information can usually be generated by extracting the visual features of the image. This involves obtaining the visual features of the image through an encoder and then using a recurrent neural network to generate an overall description of the image. However, the training process of this neural network often relies on manual annotation, which can lead to biases in the labeled data. This can result in the neural network being unable to identify the specific annotations causing problems during training, leading to a low degree of matching between the obtained image descriptive information and the content information in the image, thus reducing the accuracy of the generated image descriptive text. Summary of the Invention

[0003] Therefore, it is necessary to provide an optimized method, apparatus, computer device, and storage medium for an image description text generation model that can improve the accuracy of image description text generation, addressing the aforementioned technical problems.

[0004] Firstly, this application provides an optimization method for an image-description text generation model. The method includes:

[0005] Using image samples as input data, obtain the first word probability set output by the image description text generation model before training begins and the second word probability set output during training.

[0006] Open set recognition is performed based on image samples and the actual descriptive text represented by the second word probability set to obtain the text confidence of the actual descriptive text;

[0007] Using the first word probability set as a reference, the model loss is determined based on the second word probability set and text confidence.

[0008] Based on the model loss, the model parameters of the image description text generation model are adjusted to obtain an optimized image description text generation model.

[0009] Secondly, this application also provides an optimization apparatus for an image description text generation model. The apparatus includes:

[0010] The probability set acquisition module is used to obtain the first word probability set output by the image description text generation model before training begins and the second word probability set output during training, using image samples as input data.

[0011] The open set recognition module is used to perform open set recognition based on image samples and the actual descriptive text represented by the second word probability set, and obtain the text confidence of the actual descriptive text.

[0012] The model loss determination module is used to determine the model loss based on the second word probability set and the text confidence, with the first word probability set as a reference.

[0013] The model optimization module is used to adjust the model parameters of the image description text generation model based on the model loss, so as to obtain an optimized image description text generation model.

[0014] Thirdly, this application also provides a computer device. The computer device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to perform the following steps:

[0015] Using image samples as input data, obtain the first word probability set output by the image description text generation model before training begins and the second word probability set output during training.

[0016] Open set recognition is performed based on image samples and the actual descriptive text represented by the second word probability set to obtain the text confidence of the actual descriptive text;

[0017] Using the first word probability set as a reference, the model loss is determined based on the second word probability set and text confidence.

[0018] Based on the model loss, the model parameters of the image description text generation model are adjusted to obtain an optimized image description text generation model.

[0019] Fourthly, this application also provides a computer-readable storage medium. The computer-readable storage medium stores a computer program thereon, which, when executed by a processor, performs the following steps:

[0020] Using image samples as input data, obtain the first word probability set output by the image description text generation model before training begins and the second word probability set output during training.

[0021] Open set recognition is performed based on image samples and the actual descriptive text represented by the second word probability set to obtain the text confidence of the actual descriptive text;

[0022] Using the first word probability set as a reference, the model loss is determined based on the second word probability set and text confidence.

[0023] Based on the model loss, the model parameters of the image description text generation model are adjusted to obtain an optimized image description text generation model.

[0024] Fifthly, this application also provides a computer program product. The computer program product includes a computer program that, when executed by a processor, performs the following steps:

[0025] Using image samples as input data, obtain the first word probability set output by the image description text generation model before training begins and the second word probability set output during training.

[0026] Open set recognition is performed based on image samples and the actual descriptive text represented by the second word probability set to obtain the text confidence of the actual descriptive text;

[0027] Using the first word probability set as a reference, the model loss is determined based on the second word probability set and text confidence.

[0028] Based on the model loss, the model parameters of the image description text generation model are adjusted to obtain an optimized image description text generation model.

[0029] The aforementioned optimization method, apparatus, computer device, storage medium, and computer program product for the image description text generation model uses image samples as input data. It acquires the first word probability set output by the image description text generation model before training begins and the second word probability set output during training. Then, based on the actual description text represented by the image samples and the second word probability set, it performs open-set recognition to obtain the text confidence score of the actual description text. Next, using the first word probability set as a reference, it determines the model loss based on the second word probability set and the text confidence score. Based on the model loss, it adjusts the model parameters of the image description text generation model to obtain the optimized model. Since the confidence score is assigned to the actual description text through open-set recognition, the open-set detector can output a higher confidence score for the description text. Therefore, adjusting the parameters increases the text confidence score, which in turn improves the accuracy of the text description information. This ensures that the optimized image description text generation model can generate more accurate image description text, thus improving the accuracy of image description text generation. Attached Figure Description

[0030] Figure 1 This is an application environment diagram of an optimization method for an image description text generation model in one embodiment;

[0031] Figure 2 This is a flowchart illustrating the overall technical process of optimizing an image description text generation model in one embodiment.

[0032] Figure 3 This is a flowchart illustrating an optimization method for an image description text generation model in one embodiment;

[0033] Figure 4 This is a flowchart illustrating the optimization method for the image description text generation model in another embodiment;

[0034] Figure 5 This is a flowchart illustrating the optimization method for the image description text generation model in yet another embodiment;

[0035] Figure 6 Here is an example image of an image sample from one embodiment;

[0036] Figure 7 This is a schematic diagram of the complete process of optimizing an image description text generation model in one embodiment;

[0037] Figure 8 This is a structural block diagram of an optimization device for an image description text generation model in one embodiment;

[0038] Figure 9 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0039] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0040] With the rapid development of artificial intelligence (AI) technology, AI is being researched and applied in various fields. Image recognition technology enables "image description," which means converting the content information in an image into image descriptive information. Currently, image descriptive information is typically generated by extracting the visual features of the image. This involves an encoder acquiring the visual features of the image, followed by a recurrent neural network (RNN) to generate a comprehensive description of the image. During the training of this neural network, a training method based on human annotation preferences is often used. This means that the model can be trained based on categorical data annotations to achieve image recognition, leveraging the discrimination ability of the classification model to make it more inclined to output text correctly identified by the classification model. However, the labeled data is prone to bias, which can lead to the neural network training process being unable to identify the specific problematic annotations. This results in a low match between the obtained image descriptive information and the content information in the image. Furthermore, the generated descriptive text may contain objects not present in the image, leading to the "image illusion" problem. Secondly, training based on human preferences involves generating multiple descriptions for the same image using existing models, then manually ranking these descriptions to train the classification model. The training loss is the predicted value of all descriptions from highest to lowest. A comparative method is then used to train the model to output descriptions with higher human preferences. However, human preference generation only ensures that the text conforms to human descriptive preferences; it still cannot guarantee the accuracy of the text description word by word. Therefore, current image recognition technologies may reduce the accuracy of image description text generation.

[0041] This application provides an optimization method for an image description text generation model that improves the efficiency of such models. The optimization method provided in this application can be applied to, for example... Figure 1 In the application environment shown, terminal 102 communicates with server 104 via a network. A data storage system can store the data that server 104 needs to process. The data storage system can be integrated onto server 104, or it can be located in the cloud or on another server.

[0042] Specifically, taking server 104 as an example, server 104 uses image samples as input data to obtain the first word probability set output by the image description text generation model before training begins and the second word probability set output during training. Based on the actual description text represented by the image samples and the second word probability set, open-set recognition is performed to obtain the text confidence of the actual description text. Using the first word probability set as a reference, the model loss is determined based on the second word probability set and the text confidence. Based on the model loss, the model parameters of the image description text generation model are adjusted to obtain an optimized image description text generation model. Assigning confidence values ​​to the actual description text through open-set recognition allows the open-set detector to output higher confidence for the description text. This improves the text confidence by adjusting parameters, thereby ensuring the accuracy of the text description information. This ensures that the optimized image description text generation model can generate more accurate image description text, thus improving the accuracy of image description text generation.

[0043] The terminal 102 can be, but is not limited to, various desktop computers, laptops, smartphones, tablets, IoT devices, and portable wearable devices. IoT devices can include smart speakers, smart TVs, smart air conditioners, and smart in-vehicle devices. Portable wearable devices can include smartwatches, smart bracelets, and head-mounted devices. The server 104 can be an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms. The terminal and server can be directly or indirectly connected via wired or wireless communication, which is not limited herein. The optimization method for the image description text generation model provided in the application embodiments can be applied to various scenarios, including but not limited to cloud technology and artificial intelligence.

[0044] For ease of understanding, the overall technical process provided in the embodiments of this application is briefly described below, such as... Figure 2The overall technical flowchart shown illustrates that this application can determine the expected output result for an image sample through a combination of the following two methods, thereby completing the model optimization process. One method is optimization training using an open-set detector. Inaccurate descriptive information, especially for text that hasn't been manually adjusted, can negatively impact the image description text generation model. Therefore, this invention uses an open-set detector to detect the confidence level of the descriptive text. This detector helps determine whether there is noise in the descriptive text that doesn't match the image, resulting in a higher confidence level for the detector. Secondly, considering the potential inaccuracies in standards caused by manual adjustment, this solution, in addition to optimization training using an open-set detector, employs a manual adjustment method for auxiliary optimization training. This involves using manually adjusted descriptive text to train the image description text generation model, allowing it to learn how to adjust the output descriptive text based on the manually adjusted text. Thus, by combining these two methods, the model optimization process is completed, ensuring that the optimized image description text generation model can generate more accurate image description text, thereby improving the accuracy of image description text generation.

[0045] Furthermore, since the optimization method for the image description text generation model provided in this application embodiment also involves Artificial Intelligence (AI), AI will be introduced below. AI is a theory, method, technology, and application system that uses digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use knowledge to obtain optimal results. In other words, artificial intelligence is a comprehensive technology of computer science that attempts to understand the essence of intelligence and produce a new kind of intelligent machine that can react in a way similar to human intelligence. Artificial intelligence is also the study of the design principles and implementation methods of various intelligent machines, enabling machines to have the functions of perception, reasoning, and decision-making.

[0046] Artificial intelligence (AI) is a comprehensive discipline encompassing a wide range of fields, including both hardware and software technologies. Fundamental AI technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, pre-trained model technology, operating / interactive systems, and mechatronics. Among these, pre-trained models, also known as large-scale models or foundational models, can be widely applied to downstream tasks across various AI fields after fine-tuning. AI software technologies primarily include computer vision, speech processing, natural language processing, and machine learning / deep learning.

[0047] The solutions provided in this application relate to computer vision (CV) and machine learning (ML) technologies under the artificial intelligence (AI) framework. Computer vision is a science that studies how to enable machines to "see," specifically, it refers to machine vision that uses cameras and computers to replace human eyes for target recognition and measurement, and further performs image processing to create images more suitable for human observation or transmission to instruments for detection. As a scientific discipline, computer vision researches related theories and technologies, attempting to establish AI systems capable of extracting information from images or multidimensional data. Large model technology has brought significant changes to the development of computer vision technology. Pre-trained models in the vision field, such as Swin-transformer, ViT, V-MOE, and MAE, can be quickly and widely applied to downstream specific tasks after fine-tuning. Computer vision technology typically includes image processing, image recognition, image semantic understanding, image retrieval, OCR, video processing, video semantic understanding, video content / behavior recognition, 3D object reconstruction, 3D technology, virtual reality, augmented reality, simultaneous localization and mapping (SLAM), and other technologies, as well as common biometric recognition technologies such as facial recognition and fingerprint recognition.

[0048] Machine learning is a multidisciplinary field involving probability theory, statistics, approximation theory, convex analysis, and algorithm complexity theory, among others. It specifically studies how computers can simulate or implement human learning behavior to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their performance. Machine learning is the core of artificial intelligence and the fundamental way to endow computers with intelligence; its applications span all areas of artificial intelligence. Machine learning and deep learning typically include techniques such as artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and pre-trained learning. Pre-trained models represent the latest development in deep learning, integrating all of these techniques.

[0049] The following examples illustrate this in detail: In one example, as... Figure 3 As shown, an optimization method for an image description text generation model is provided, which is then applied to... Figure 1 Taking server 104 as an example, it can be understood that this method can also be applied to terminal 102 with model training computing power, and can also be applied to a system including terminal 102 and server 104, and implemented through the interaction between terminal 102 and server 104. In this embodiment, the method includes the following steps:

[0050] Step 302: Using image samples as input data, obtain the first word probability set output by the image description text generation model before training begins and the second word probability set output during training.

[0051] Among them, the first word probability set is the result obtained by generating a description text for an image sample before the start of training of the image description text generation model. The image description text generation model before the start of training is a model that does not perform training optimization and has its model parameters frozen. Therefore, the first word probability set obtained will not be adjusted or changed based on the process of model training optimization. Also, during the process of generating a description text for an image sample, the marking production of the text is to divide the to-be-generated description text into multiple to-be-generated characters, and then by looking up which character in the look-up table the to-be-generated character specifically belongs to [Token], and then determining the to-be-generated character as the matched character to obtain the actually generated character. Then, how to find the character matched by the to-be-generated character, that is, it is necessary to generate the output probability of each to-be-generated character that makes up the description text belonging to each character in the character table. Thus, through the aforementioned probabilities, the description text for the image sample can be obtained. Therefore, the first word probability set is used to represent: the first word probability set of each to-be-generated character belonging to each character in the character table.

[0052] It can be understood that the aforementioned characters can specifically be words, symbols, etc., such as: the word "one", the word "good", the symbol ",", and the character set also includes an end character to indicate the end of probability generation. That is, in practical applications, the output probability of each to-be-generated character that makes up the description text belonging to each character in the character table is generated in sequence, and when the output probability of the to-be-generated character belonging to the end character is the maximum value, ending the subsequent probability output can obtain the description text for the image sample.

[0053] Similarly, the second word probability set is the result obtained by generating a description text for an image sample during the training optimization process of the image description text generation model. The image description text generation model during the training process will optimize the parameters through the obtained model loss. Therefore, the image description text generation model during the training process performs normal training and saves the parameters after each adjustment. And the second word probability set is used to represent: the actual output probability of each to-be-generated character belonging to each character in the character table.

[0054] Specifically, in the case where it is necessary to optimize the parameters of the image description text generation model, before the start of model training, the image sample is used as the input of the image description text generation model to obtain the first word probability set output by the image description text generation model before the start of training. And during the process of starting to train and optimize the image description text generation model, the image description text generation model will output the second word probability set.

[0055] In one specific embodiment, using image samples as input data, the process of obtaining a first word probability set output by the image description text generation model before training begins and a second word probability set output during training includes: before training the image description text generation model begins, inputting image samples into the model to obtain the initial output probability of the character to be generated belonging to each character in the character list, and determining the initial output probability of the character to be generated belonging to each character in the character list as the first word probability set; during training the image description text generation model, inputting image samples into the model to obtain the actual output probability of the character to be generated belonging to each character in the character list, and determining the actual output probability of the character to be generated belonging to each character in the character list as the second word probability set.

[0056] Before training the image-to-text generation model, image samples are input into the model to obtain the initial output probability of the character to be generated belonging to each character in the word list. This initial output probability is then defined as the first word probability set, which characterizes the initial output probability of the character to be generated belonging to each character in the word list. Since the first word probability set characterizes the initial output probability of the character to be generated belonging to each character in the word list, and the initial output probabilities are generated sequentially—that is, before outputting the initial description text, the initial output probability is output for each character to be generated constituting the initial description text. At this point, the character to be generated is the ungenerated empty character [empty]. After completing the initial output probability of a character to be generated belonging to each character in the word list, the model selects which character in the word list the character to be generated belongs to. This results in the determined character to be generated (i.e., the initial generated character). Then, the initial generated character is used to predict the next adjacent character to be generated, thus obtaining the initial output probability of the next character to be generated.

[0057] Similarly, during the training of the image-description text generation model, image samples are input into the model to obtain the actual output probability of the character to be generated belonging to each character in the word list. This actual output probability is defined as the second word probability set, which is then used to determine the actual output probability of the character to be generated belonging to each character in the word list. Based on this, in a specific embodiment, during the training of the image-description text generation model, inputting image samples into the model to obtain the actual output probability of the character to be generated belonging to each character in the word list includes: during the training of the image-description text generation model, inputting image samples into the model sequentially generates the actual output probability of the character to be generated belonging to each character in the word list that constitutes the actual description text; when the actual output probability of the character to be generated belonging to the end character in the word list reaches its maximum value, the output of the actual output probability is stopped to obtain the actual output probability of the character to be generated belonging to each character in the word list.

[0058] The second word probability set is used to represent the actual output probability of the character to be generated belonging to each character in the word table, and the actual output probability is generated sequentially. That is, during the training process of the image description text generation model, the server inputs image samples into the image description text generation model. Before generating the actual description text, it outputs the corresponding actual output probability for the characters to be generated that constitute the actual description text. At this time, the characters to be generated are empty characters [empty] that have not yet been generated. After completing the actual output probability of a character to be generated belonging to each character in the word table, the system selects which character in the word table the character to be generated belongs to. At this time, the determined character to be generated (i.e., the actual generated character) is obtained. Then, the actual generated character is used to predict the next adjacent character to be generated to obtain the actual output probability corresponding to the next character to be generated. This continues until the actual output probability of the character to be generated belonging to the end character is at its maximum value, at which point the acquisition of the actual output probability stops, thus obtaining the actual output probability of the character to be generated belonging to each character in the word table.

[0059] In another specific embodiment, when the image description text generation model outputs the first word probability set before training begins, the model parameters are in a frozen state, meaning the model parameters will not be fine-tuned at this time, and therefore the obtained first word probability set will not change. Conversely, when the image description text generation model outputs the second word probability set during training, the model parameters are in an adjustable state, meaning the second word probability set may change accordingly based on changes in the model parameters during training. Therefore, in each training iteration, image samples are used as input to the image description text generation model, resulting in the second word probability set for each image sample.

[0060] Step 304: Perform open set recognition based on the image samples and the actual descriptive text represented by the second word probability set to obtain the text confidence of the actual descriptive text.

[0061] Since the second word probability set specifically represents the actual output probability of the character to be generated belonging to each character in the word table, the actual descriptive text represented by the second word probability set is determined by the actual output probability of the character to be generated belonging to each character in the word table. The following describes how to obtain the actual descriptive text represented by the second word probability set. In a specific embodiment, the optimization method for the image descriptive text generation model further includes: determining the probability generation order of each actual output probability in the second word probability set; determining the character matched by the maximum value of the actual output probability of the character to be generated belonging to each character in the word table as the actual generated character; and concatenating the actual generated characters sequentially based on the probability generation order to obtain the actual descriptive text.

[0062] Secondly, Open Set Recognition uses a model trained on a training set. When tested with a test set (containing categories not found in the training set), if the input data has a known category, it outputs the specific category; if the input data has an unknown category, it performs appropriate processing (identifying it as "unknown"). Open Set Recognition identifies the confidence level of descriptive text, which specifically represents the probability that the information described in the descriptive text exists in the image. Therefore, text confidence level represents the probability that the noun (i.e., the item) described in the actual descriptive text exists in the image sample.

[0063] In this embodiment, the open-set detector can be a grounding dino or an APE, etc. The open-set detector takes a natural language description and its corresponding image as input, performs open-set detection, and outputs a confidence score for the natural language description. Essentially, it's a model that aligns the natural language description and the image's features to calculate similarity; low similarity indicates the absence of the description, while high similarity indicates its presence and is detected. Using an open-set detector can help determine if there is noise in the natural language description that doesn't match the image.

[0064] Specifically, during the training process of the image description text generation model, the server determines the actual description text by using the actual output probability of the character to be generated belonging to each character in the word table, represented by the second word probability set. That is, the server first predicts the actual output probability of the first character to be generated belonging to each character in the word table, then determines the character that matches the maximum value of the actual output probability of the character to be generated belonging to each character in the word table as the actual generated character, and then predicts the next adjacent character to be generated using the determined actual generated character, until the actual output probability of the character to be generated belonging to the end character is at its maximum value, at which point the acquisition of actual output probabilities stops, and the probability generation order of each actual output probability is determined. The actual generated characters are then concatenated in the order of probability generation to obtain the actual description text.

[0065] For ease of understanding, taking a word list containing characters B1 to B100 as an example, the second word probability set represents the actual output probability of each character to be generated belonging to characters B1 to B100. Thus, we can obtain the actual output probability of the first character to be generated, C1, belonging to characters B1 to B100. If it is determined that the actual output probability of the character to be generated, C1, belonging to character B4 is the highest, then we can determine that the actual generated character D1 corresponding to character C1 is specifically character B4. Then, we can continue to output the actual output probability of the next character to be generated, C2, belonging to characters B1 to B100. If it is determined that the actual output probability of the character to be generated, C2, belonging to character B21 is the highest, then we can determine that the actual generated character D2 corresponding to character C2 is specifically character B21.

[0066] Similarly, continue to output the actual output probabilities that the next character to be generated C3 corresponding to the actually generated character D2 of the character C2 to be generated belongs to each of the characters B1 to B100. If it is determined that the actual output probability that the character C3 to be generated belongs to the character B7 is the largest, at this time, it can be determined that the actually generated character D3 corresponding to the character C3 to be generated is specifically the character B7. And then output the actual output probabilities that the next character to be generated C4 corresponding to the actually generated character D3 of the character C3 to be generated belongs to each of the characters B1 to B100. If it is determined that the actual output probability that the character C4 to be generated belongs to the character B15 is the largest, at this time, it can be determined that the actually generated character D4 corresponding to the character C4 to be generated is specifically the character B15. If after confirming that the actually generated character D4 corresponding to the character C4 to be generated is specifically the character B15, the actual output probability that the next character to be generated belongs to the end character is the largest, stop obtaining the actual output probabilities at this time. At this time, it can be determined that the probability generation order is the actually generated character D1, the actually generated character D2, the actually generated character D3, and the actually generated character D4. Then, connect the actually generated characters D1, D2, D3, and D4 in sequence based on the foregoing probability generation order, that is, connect the specific character B4 of the actually generated character D1, the specific character B21 of the actually generated character D2, the specific character B7 of the actually generated character D3, and the character B15 of the actually generated character D4 in sequence, thereby forming the actual description text.

[0067] For example, if among the actual output probabilities that the first character to be generated belongs to each character in the character table, the actual output probability that it belongs to the character [big] is the largest, at this time, it will be determined that the actually generated character of the first character to be generated is [big]. Then, continue to output the actual output probabilities that the next character to be generated belongs to each character in the character table in sequence. If the actual output probability that it belongs to the character [sea] is the largest, at this time, it will be determined that the actually generated character of the next character to be generated is [sea]. And so on. If the actually generated characters of the subsequent characters to be generated determined in sequence are [and] and [ship], and after confirming the actually generated character [ship], the actual output probability that it belongs to the end character is the largest, stop obtaining the actual output probabilities at this time. At this time, splice the actually generated characters in sequence according to the probability generation order to obtain the actual description text as "big sea and ship".

[0068] Furthermore, since open-set detectors cannot directly detect whether the actual descriptive text is correct, and open-set detectors may also have inaccurate detection capabilities, this embodiment optimizes the training by re-assigning weights (i.e., confidence scores) to the actual descriptive text. That is, the server uses the actual descriptive text and image samples as input to the open-set detector, which then outputs the text confidence score of the actual descriptive text. It is understandable that, since open-set detectors can only detect whether nouns are accurate, in practical applications, if the actual descriptive text does not contain nouns, an open-set recognition step should be performed on the actual descriptive text, and other training methods should be used instead.

[0069] Therefore, the server first needs to determine if the actual descriptive text contains the term (or object) to be detected. Then, it performs open-set recognition on the actual descriptive text containing the term and the image sample, and determines the text confidence of the actual descriptive text based on the noun confidence obtained for the term to be detected. For example, if the probability of open-set recognition detecting "bird" in the image sample is 0.8, then the text confidence of the actual descriptive text is 0.8 if the actual descriptive text contains the term to be detected. If the actual descriptive text contains multiple terms to be detected, then the noun confidence of each term to be detected is obtained first through open-set recognition, and then the noun confidence of each term to be detected is averaged. The averaged noun confidence is then determined as the text confidence of the actual descriptive text. For example, if the probability of open-set recognition detecting "ocean" in the image sample is 0.9, and the probability of detecting "person" is 0.3, then the text confidence of the actual descriptive text is 0.6 [(0.9+0.3) / 2].

[0070] Step 306: Using the first word probability set as a reference, determine the model loss based on the second word probability set and text confidence.

[0071] Specifically, since the goal of model optimization is to make the image description text generation model, after training and optimization, better than the image description text generation model before training and optimization, the model training process should learn accurate descriptive information and how to adjust descriptive information with errors. Therefore, when determining the model loss, the server should use the first word probability set obtained by the un-optimized image description text generation model as a reference to avoid outputting results that are completely semantically incorrect with the first word probability set. Instead, it should perform positive optimization based on the image description text generation model, and then determine the model loss based on the second word probability set and text confidence. This ensures that the image description text generation model trained and optimized based on this model loss, without deviating in the opposite direction from the first word probability set, adjusts the second word probability set output by the model based on the expected output result to achieve the purpose of model optimization.

[0072] In one specific embodiment, the model loss is determined based on the second word probability set and the text confidence, using the first word probability set as a reference, including: calculating the consistency loss based on the first word probability set and the second word probability set; and determining the model loss using the text confidence and the consistency loss.

[0073] Considering that the model might output semantically inaccurate but noun-repeating text to increase confidence during training and tuning, a consistency loss needs to be added to the loss calculation. This consistency loss ensures the consistency between the second and first word probability sets, describing the differences between them. Specifically, the derivative of the actual output probability of the character to be generated belonging to each character in the word table, represented by the second word probability set, is calculated to obtain the first derivative. The derivative of the initial output probability of the character to be generated belonging to each character in the word table, represented by the first word probability set, is calculated to obtain the second derivative. The difference between the first and second derivatives is then multiplied by a preset parameter to obtain the consistency loss.

[0074] Based on this, the server determines the model loss by using text confidence and consistency loss. That is, the server calculates the difference between text confidence and consistency loss, and uses the result as the model loss. For easier understanding of the aforementioned processing method, see formula (1):

[0075] (1)

[0076] in, Indicates the model loss. Indicates text confidence level. Indicates consistency loss. Represents the probability set of the second word. Represents the probability set of the first word. This indicates the preset parameters.

[0077] Step 308: Based on the model loss, adjust the model parameters of the image description text generation model to obtain an optimized image description text generation model.

[0078] Specifically, the server adjusts the model parameters of the image description text generation model based on the model loss to obtain an optimized image description text generation model. That is, after calculating the model loss, the server determines whether the loss function of the image description text generation model has reached the convergence condition. If it has not reached the convergence condition, the server adjusts the model parameters of the image description text generation model using the model loss. As mentioned earlier, the adjusted model parameters are saved during training. Therefore, when the loss function of the image description text generation model reaches the convergence condition, the optimized image description text generation model is obtained based on the saved model parameters after the last adjustment. In practical applications, this optimized model is used to generate descriptive text for images.

[0079] The convergence condition for the aforementioned loss function can be that the loss value is less than or equal to a first preset threshold. For example, the first preset threshold could be 0.005, 0.01, 0.02, or other values ​​close to 0. Alternatively, the convergence condition can be that the difference between two consecutive loss values ​​is less than or equal to a second preset threshold. The second threshold can be the same as or different from the first threshold. For example, the second preset threshold could be 0.005, 0.01, 0.02, or other values ​​close to 0. Another convergence condition could be that the number of times the model parameters of the image-text generation model are updated reaches an update iteration threshold. In practical applications, other convergence conditions can also be used, which are not limited here.

[0080] It should be understood that the corresponding examples in the embodiments of this application are used to understand this solution, but should not be construed as specific limitations on this solution.

[0081] In the optimization method of the image description text generation model described above, image samples are used as input data. The model obtains the first word probability set output before training and the second word probability set output during training. Then, open-set recognition is performed based on the actual description text represented by the image samples and the second word probability set to obtain the text confidence score of the actual description text. Using the first word probability set as a reference, the model loss is determined based on the second word probability set and the text confidence score. Based on the model loss, the model parameters of the image description text generation model are adjusted to obtain the optimized model. Since the confidence score is assigned to the actual description text through open-set recognition, the open-set detector can output a higher confidence score for the description text. Therefore, adjusting the parameters increases the text confidence score. This increased confidence score ensures the accuracy of the text description information, guaranteeing that the optimized image description text generation model can generate more accurate image description text, thus improving the accuracy of image description text generation.

[0082] Understandably, if open-set detection is performed directly on the descriptive text, since the descriptive text may contain many nouns, and in practical applications, the descriptive text usually consists of more than one sentence, then when the descriptive text includes multiple sentences, the sentences of the descriptive text can be used as a more granular detection dimension for open-set identification, ensuring that the obtained confidence scores are more accurate. Based on this, in one embodiment, such as Figure 4 As shown, optimization methods for image description text generation models also include:

[0083] Step 402: Segment the actual description text to obtain multiple actual description sentences.

[0084] Specifically, the server segments the actual descriptive text into multiple descriptive sentences. Sentence segmentation can be performed through semantic text segmentation or other methods; no specific method is specified here. For ease of understanding, let's take the actual descriptive text "This is a picture of the sea, there is a small platform boat in the sea, and there are people on the small platform boat" as an example. This would result in the descriptive sentences "This is a picture of the sea," "There is a small platform boat in the sea," and "There are people on the small platform boat."

[0085] Based on this, open-set recognition is performed on the image samples and the actual descriptive text represented by the second word probability set to obtain the text confidence of the actual descriptive text, including:

[0086] Step 404: Perform open set recognition on the image samples and each actual descriptive sentence to obtain the actual sentence confidence of each actual descriptive sentence; the text confidence includes the actual sentence confidence of each actual descriptive sentence.

[0087] Specifically, the server uses image samples and each actual descriptive sentence as input to the open-set detector. The open-set detector outputs the actual sentence confidence score of each actual descriptive sentence relative to the image sample. The method for obtaining the actual sentence confidence score is similar to that for obtaining the actual confidence score, and will not be repeated here. It is understood that open-set detection only applies to actual descriptive text containing the term to be detected, or to actual descriptive sentences containing the term to be detected. If the term to be detected is not included, the expected output result is not determined based on that actual descriptive sentence or actual descriptive text, nor is subsequent loss calculation performed.

[0088] Based on this, in an optional embodiment, open set identification is performed on the image sample and each actual descriptive sentence to obtain the actual sentence confidence of each actual descriptive sentence, including: determining open set descriptive sentences that include nouns from the actual descriptive sentences; and performing open set identification on the image sample and each open set descriptive sentence to obtain the actual sentence confidence of each open set descriptive sentence.

[0089] Specifically, the server first needs to determine whether there is a noun (or object) to be detected in each actual description sentence. If so, the actual description sentence is determined to be an open set description sentence. If not, the server performs open set recognition on the actual description sentences that do not contain nouns. Based on this, the server then performs open set recognition on the actual description sentences (open set description sentences) that contain nouns to be detected and the image samples to obtain the actual sentence confidence of each open set description sentence.

[0090] To facilitate understanding, let's take the following descriptive sentences as examples: "This is a picture of the sea," "There is a small platform boat in the sea," "There are people on the small platform boat," and "How beautiful!" An open-set detector can identify the probability of the presence of "sea," "small platform boat," and "people" in the image sample. Based on these probabilities, the confidence level of each descriptive sentence is determined. Since the descriptive sentence "How beautiful!" does not contain any nouns, it is not subject to open-set detection. The results of open-set detection for the other descriptive sentences are as follows: the confidence level of "This is a picture of the sea" is 90%, the confidence level of "There is a small platform boat in the sea" is 80%, and the confidence level of "There are people on the small platform boat" is 40%.

[0091] Based on this, using the first word probability set as a reference, and based on the second word probability set and text confidence, the model loss is determined, including:

[0092] Step 406: Average the confidence scores of each actual descriptive clause to obtain the average confidence score.

[0093] Specifically, the server averages the confidence scores of each actual sentence to obtain the average confidence score. That is, the server determines whether to segment the actual descriptive text into sentences, and can also determine the number of sentences in multiple actual descriptive sentences. This number of sentences specifically refers to the number of open-set descriptive sentences. Then, the confidence scores of each actual descriptive sentence are summed to obtain the summed confidence score. The average confidence score is obtained by dividing the summed confidence score by the number of sentences. For example, the actual sentence confidence score of the descriptive sentence "This is a picture of the sea" is 90%, the actual sentence confidence score of the descriptive sentence "There is a platform boat in the sea" is 80%, and the actual sentence confidence score of the descriptive sentence "There are people on the platform boat" is 40%. In this case, the average confidence score is 70% [(90% + 80% + 40%) / 3].

[0094] Step 408: Calculate the consistency loss based on the probability set of the first word and the probability set of the second word.

[0095] Specifically, the actual output probability of the character to be generated belonging to each character in the word table, represented by the second word probability set, is differentiated to obtain the first derivative result. The initial output probability of the character to be generated belonging to each character in the word table, represented by the first word probability set, is differentiated to obtain the second derivative result. The difference between the first and second derivative results is calculated, and the difference result is multiplied by a preset parameter to obtain the consistency loss.

[0096] Step 410: Determine the model loss using the average confidence level and the consistency loss.

[0097] Specifically, the server determines the model loss by subtracting the average confidence score from the consistency loss. That is, the server calculates the difference between the average confidence score and the consistency loss, and uses the result as the model loss. For easier understanding of the aforementioned processing method, see formula (2):

[0098] (2)

[0099] in, Indicates the model loss. This represents the average confidence level. Indicates consistency loss. Represents the probability set of the second word. Represents the probability set of the first word. Indicates preset parameters. This indicates the actual clause confidence level of the actual descriptive clause i. This indicates the number of descriptive clauses that can be opened sets.

[0100] It should be understood that the corresponding examples in the embodiments of this application are used to understand this solution, but should not be construed as specific limitations on this solution.

[0101] In this embodiment, considering that the model may output semantically inaccurate but noun-repeated text to increase confidence during training and tuning, a consistency loss is added to maintain consistency, that is, to ensure the consistency between the probability set of the second word and the probability set of the first word. Then, the confidence of each sentence is considered to ensure that the descriptive text with higher confidence can be output. This further ensures that the model is tuned as a whole to improve the accuracy of image descriptive text generation based on the optimized model.

[0102] Furthermore, as discussed above, this solution considers the potential inaccuracies in standards that may result from manual adjustments. Therefore, in addition to optimizing the training using an open-set detector, it employs manual adjustments for auxiliary optimization training. In other words, this application combines the use of an open-set detector to detect the confidence level of the descriptive text with manual adjustments for auxiliary optimization training to jointly optimize the model. A detailed description follows, in one embodiment, such as... Figure 5 As shown, optimization methods for image description text generation models also include:

[0103] Step 502: Determine the initial description text represented by the first word probability set, and determine the adjusted description text based on the initial description text.

[0104] Specifically, as discussed above, the first word probability set represents the initial output probability of the character to be generated belonging to each character in the word table. Specifically, it sequentially generates the initial output probability of each character to be generated constituting the initial description text belonging to each character in the word table. That is, before outputting the initial description text, the corresponding initial output probability is output for each character to be generated constituting the initial description text. At this point, the character to be generated is [empty]. After outputting the initial output probability of the first character to be generated belonging to each character in the word table, the selection process determines which character in the word table the character to be generated belongs to. This results in the determined character to be generated (i.e., the initial generated character). Then, the prediction is performed on the next character to be generated adjacent to the initial generated character until the initial output probability of the character to be generated belonging to the end character reaches its maximum value, at which point the acquisition of the initial output probability stops.

[0105] Furthermore, the server determines the probability generation order of each initial output probability in the first word probability set, and then determines the character that matches the maximum value of the initial output probability of each character in the word table as the initial generated character. Based on the probability generation order, the initial generated characters are concatenated sequentially to obtain the initial description text. It is understood that the method of generating the initial description text is similar to that of the actual description text described in the previous embodiments, and will not be repeated here.

[0106] Based on this, the adjusted description text is determined based on the initial description text. Specifically, the initial description text is adjusted, and in this embodiment, this adjustment is performed manually. Specifically, the initial description text is adjusted manually to obtain the adjusted description text. Based on this, the adjustment of the initial description text follows three adjustment principles: text correction, text deletion, and text filling. Text correction involves modifying erroneous information in the initial description text regarding the image samples. Text deletion involves removing redundant information in the initial description text regarding the image samples. And text filling involves supplementing missing information in the initial description text regarding the image samples. These three adjustment principles ensure that the image description text generation model can generate sufficiently detailed description text while correcting various descriptive problems such as errors, redundancy, or omissions. In other words, adjusting the initial description text can specifically involve at least one of text correction, text deletion, and text filling.

[0107] For ease of understanding, image samples are used as follows: Figure 6 Taking the image sample shown as an example, Figure 6 The image sample shown depicts a vast ocean with a platform and small boats, and clouds in the distance. If the image description text generation model is designed for… Figure 6 The initial description text for the image sample shown is "This is a picture of the sea, and there are several people swimming on the beach." However, the image sample actually shows no one swimming. Therefore, the initial description text "This is a picture of the sea, and there are several people swimming on the beach" needs to be adjusted. First, text deletion is performed to remove the erroneous description "There are several people swimming on the beach." Since the image sample also contains platforms, small boats, and clouds in the sea, text filling is also necessary to add the text description "There are platforms and small boats in the sea, and clouds in the distance." Based on this, the adjusted description text "This is a picture of the sea, and there are several people swimming on the beach" becomes "This is a picture of the sea, and there are platforms and small boats in the sea, and clouds in the distance."

[0108] Step 504: Obtain the probability set of the third word output by the image description text generation model before training begins, with the adjusted description text as the output target, and the probability set of the fourth word output during training, with the adjusted description text as the output target.

[0109] Specifically, the server needs to obtain the third word probability set output by the image description text generation model before training begins, with the adjusted description text as the output target. This third word probability set includes the initial character probabilities of the characters to be generated belonging to the adjusted characters in the character set used to construct the adjusted description text. As mentioned earlier, since the adjusted description text of the image sample is the target of model tuning, the server first determines the order and number of adjusted characters corresponding to the adjusted characters included in the adjusted description text. The number of adjusted characters includes the number of characters, symbols, and the number of ending characters in the adjusted description text. For ease of understanding, let's take the adjusted description text "This is a picture of the sea, with platforms and small boats in the sea, and clouds in the distance" as an example. This adjusted description text can be divided into 25 characters: [this], [is], [one], [picture], [sea], [of], [picture], [picture], [,], [big], [sea], [in], [have], [platform], [small], [boat], [,], [far], [place], [still], [have], [clouds], . That is, the number of characters adjusted is 25, and the order of the characters is as described above.

[0110] Based on this, before the image description text generation model begins training—that is, using an image description text generation model without model parameter adjustments—initial character probabilities are generated sequentially according to the order of the adjusted characters, assigning each character to a given character in the character list as its initial character probability. Then, from these initial character probabilities, the initial character probability corresponding to the adjusted character is selected. It can be understood that the initial character probability corresponding to the adjusted character can be the maximum value among the initial character probabilities corresponding to the characters to be generated, but it must correspond to the adjusted character. This process continues until the number of initial character probabilities corresponding to the adjusted characters reaches the number of adjusted characters, thus obtaining the initial character probabilities corresponding to each adjusted character that constitutes the adjusted description text. These initial character probabilities are then defined as the third word probability set.

[0111] For ease of understanding, again take the example of adjusting the descriptive text "This is a picture of the sea, there are platform boats in the sea, and there are also clouds in the distance", that is, the image description text generation model without adjusting the model parameters will first output the initial character probabilities of each character in the vocabulary for the first character to be generated. Then, from the initial character probabilities of each character in the vocabulary for the first character to be generated, the initial character probability corresponding to the character [这] is selected as the initial character probability corresponding to the adjustment character. After determining the character [这], the initial character probabilities of each character in the vocabulary for the next character to be generated after the character [这] are output in a similar manner. Then, from the initial character probabilities of each character in the vocabulary for the next character to be generated after the character [这], the initial character probability corresponding to the character [是] is selected as the initial character probability corresponding to the adjustment character. And so on. Finally, the initial character probability corresponding to the character [云] can be obtained as the initial character probability corresponding to the adjustment character. At this time, the initial character probabilities corresponding to 24 adjustment characters are generated. Finally, the initial character probabilities of each character in the vocabulary for one more character to be generated will be generated. Regardless of whether the character corresponding to the maximum value of the finally generated initial character probability is the end , since the number of adjustment characters "25" has been reached, the acquisition of the initial character probabilities will stop, and the initial character probabilities corresponding to "[这], [是], [一], [张], [大], [海], [的], [图], [片], [,], [大], [海], [中], [有], [平], [台], [小], [船], [,], [远], [处], [还], [有], [云]" are determined as the third word probability set.

[0112] Furthermore, the fourth word probability set includes the actual character probabilities of the characters to be generated belonging to the adjusted characters in the word table that construct the adjusted descriptive text. That is, the server first determines the order and number of adjusted characters corresponding to the adjusted characters included in the adjusted descriptive text. The number of adjusted characters includes the number of characters, symbols, and the number of ending characters . Then, during the training process of the image descriptive text generation model, the model, whose parameters can be adjusted, generates the actual character probabilities of the characters to be generated belonging to each character in the word table according to the adjusted character order. Then, from the actual character probabilities of the characters to be generated belonging to each character in the word table, the actual character probabilities corresponding to the adjusted characters are selected. It can be understood that the actual character probability corresponding to the adjusted character can be the maximum value among the actual character probabilities corresponding to the characters to be generated, but it must correspond to the adjusted character, until the number of actual character probabilities corresponding to the adjusted characters reaches the number of adjusted characters, thus obtaining the actual character probabilities corresponding to each adjusted character constituting the adjusted descriptive text. The actual character probabilities corresponding to each adjusted character are determined as the fourth word probability set. The method for obtaining the fourth word probability set is similar and will not be repeated here.

[0113] Based on this, using the first word probability set as a reference, and based on the second word probability set and text confidence, the model loss is determined, including:

[0114] Step 506: Using the first word probability set as a reference, determine the first model loss based on the second word probability set and text confidence.

[0115] Specifically, the server uses the first word probability set as a reference and determines the first model loss based on the second word probability set and text confidence. That is, the server can calculate the consistency loss based on the first word probability set and the second word probability set, and then determine the first model loss through the text confidence and consistency loss. The first model loss is obtained through the aforementioned embodiment and formula (1), which will not be repeated here.

[0116] Secondly, considering the segmentation of the text, the actual descriptive text can be segmented to obtain multiple actual descriptive sentences. Then, open set recognition is performed on the image sample and each actual descriptive sentence to obtain the confidence of each actual descriptive sentence. Then, the confidence of each actual descriptive sentence is averaged to obtain the average confidence. The consistency loss is calculated based on the first word probability set and the second word probability set. The model loss is determined by the average confidence and the consistency loss. Specifically, the first model loss is obtained through the aforementioned embodiment and formula (2), which will not be elaborated here.

[0117] Step 508: Using the first and third word probability sets as references, and based on the second and fourth word probability sets, determine the second model loss.

[0118] Specifically, since the goal of model optimization is to make the image description text generation model, after training and optimization, better than the model before training and optimization, the model training process should learn accurate descriptive information and how to adjust descriptive information with errors. Therefore, when determining the model loss, the server should use the first and third word probability sets obtained by the un-optimized model as a reference to avoid outputting results that are completely semantically incorrect with the first word probability set, and also to ensure that the obtained text is closer to the third word probability set. Thus, the model loss is then determined based on the second and fourth word probability sets to ensure that the image description text generation model trained and optimized based on this model loss, without deviating negatively from the first word probability set, can learn how to adjust to be closer to the third word probability set, thereby achieving the goal of model optimization.

[0119] In one specific embodiment, the second model loss is determined based on the first word probability set and the third word probability set, and the second word probability set and the fourth word probability set. This includes determining the second model loss based on the ratio between the first word probability set and the second word probability set, and the ratio between the third word probability set and the fourth word probability set.

[0120] Specifically, the server determines the model loss based on the ratio between the probability sets of the third and fourth words and the ratio between the probability sets of the first and second words. That is, the server performs logarithmic processing on the ratio between the third and fourth word probabilities to obtain the first logarithmic value, and then multiplies the first logarithmic value with the balance parameter to obtain the first balance loss. Similarly, the server performs logarithmic processing on the ratio between the first and second word probabilities to obtain the second logarithmic value, and then multiplies the second logarithmic value with the balance parameter to obtain the second balance loss. Finally, the model loss is determined based on the difference between the first and second balance losses. For easier understanding of the aforementioned processing method, see formula (3):

[0121] (3)

[0122] in, Indicates the model loss. Represents the probability set of the fourth word. Represents the probability set of the third word. Represents the probability set of the second word. Represents the probability set of the first word. Represents the balance parameters. This represents the value processed by the first logarithm. This represents the value processed by the second logarithm. This represents the first equilibrium loss. This represents the second balancing loss. The balancing parameter is typically set to 0.1.

[0123] Understandably, if preference training is performed directly on the descriptive text, the model will learn less information during optimization because the initial descriptive text and the adjusted descriptive text have similar overall word order and sentence structure. Furthermore, in practical applications, descriptive text usually consists of more than one sentence. When descriptive text includes multiple sentences, the desired output can be determined from two dimensions: the entire descriptive text and its individual sentences. In an optional embodiment, the optimization method for the image descriptive text generation model further includes: segmenting the initial descriptive text to obtain multiple initial descriptive sentences, and segmenting the adjusted descriptive text to obtain multiple adjusted descriptive sentences.

[0124] Specifically, the server segments the initial description text into multiple initial description sentences. Similarly, the server also segments the adjusted description text into multiple adjusted description sentences. Sentence segmentation can be performed through text semantic segmentation or other segmentation methods; no specific method is specified here. For ease of understanding, we will use a previous example: segmenting the initial description text "This is a picture of the sea, and there are several people swimming on the beach" yields the initial description sentences "This is a picture of the sea" and "There are several people swimming on the beach." Similarly, segmenting the adjusted description text "This is a picture of the sea, there are platforms and small boats in the sea, and clouds in the distance" yields the adjusted description sentences "This is a picture of the sea," "There are platforms and small boats in the sea," and "Clouds in the distance."

[0125] Based on this, since descriptive text usually consists of more than one sentence, when descriptive text includes multiple clauses, the expected output can be determined from two dimensions: the whole sentence of the descriptive text and the clauses of the descriptive text. Therefore, when determining the model loss, which includes the adjusted descriptive text and multiple adjusted descriptive clauses, the model loss is also calculated from two dimensions: the whole sentence of the descriptive text and the clauses of the descriptive text.

[0126] In one specific embodiment, the optimization method of the image description text generation model further includes: determining the initial description sentence probability of each initial description sentence from the first word probability set, and determining the initial adjusted sentence probability of each adjusted description sentence from the third word probability set; determining the actual description sentence probability of each actual description sentence from the second word probability set, and determining the actual adjusted sentence probability of each adjusted description sentence from the fourth word probability set.

[0127] Specifically, before the image description text generation model begins training, the server first obtains a first word probability set and a third word probability set for the entire text sentence dimension, using a method similar to the aforementioned embodiments. For the text sentence dimension, it determines the initial description sentence probability of each initial description sentence from the first word probability set and the initial adjusted description sentence probability from the third word probability set. Then, during the training of the image description text generation model, it first obtains a second word probability set and a fourth word probability set for the entire text sentence dimension, using a method similar to the aforementioned embodiments. It then determines the actual description sentence probability of each actual description sentence from the second word probability set and the actual adjusted description sentence probability from the fourth word probability set.

[0128] Since the first and second word probability sets are text-dimensional probability sets directly output by the model, the probabilities corresponding to the characters constituting each sentence can be extracted from these sets. The third and fourth word probability sets, however, are output probabilities based on the adjusted descriptive text as the output target. Therefore, taking the third word probability set as an example, this section will explain how to obtain the third word probability set and how to extract the initial adjusted sentence probabilities from it.

[0129] 1. Obtaining the third word probability set for the entire text sentence: The server first determines the order and number of adjusted characters corresponding to the adjusted characters included in the adjusted description text. Then, using the image description text generation model without model parameter adjustment, it generates the character probability of the character to be generated belonging to each character in the character table in the order of the adjusted characters. Then, it selects the character probability corresponding to the adjusted character from the character probability of the character to be generated belonging to each character in the character table as the third word probability set of the adjusted character, until the number of generated third word probability sets reaches the number of adjusted characters.

[0130] 2. Extract the initial adjusted clause probability from the third-word probability set for the dimension of text clause segmentation: The server first determines the clause adjustment characters included in the adjusted description clause, the corresponding clause adjustment character order and the number of clause adjustment characters respectively. The number of clause adjustment characters is similar to the number of adjustment characters. For ease of understanding, take the adjusted description clause "This is a picture of the sea", the adjusted description clause "There are platforms and small boats in the sea", and the adjusted description clause "There are also clouds in the distance" as examples. The adjusted description clause "This is a picture of the sea" can be divided into 10 characters: [This], [is], [a], [picture], [of], [the], [sea], , that is, the corresponding number of clause adjustment characters is 10, and the clause adjustment character order is the sequential order of the aforementioned arrangement. Similarly, the adjusted description clause "There are platforms and small boats in the sea" can be divided into 9 characters: [the], [sea], [there], [are], [platforms], [and], [small], [boats], , that is, the corresponding number of clause adjustment characters is 9, and the clause adjustment character order is the sequential order of the aforementioned arrangement. The adjusted description clause "There are also clouds in the distance" can be divided into 6 characters: [in], [the], [distance], [there], [are], [also], [clouds], , that is, the corresponding number of clause adjustment characters is 6, and the clause adjustment character order is the sequential order of the aforementioned arrangement.

[0131] Based on this, through the image description text generation model without model parameter adjustment, for each adjusted description clause, generate the character probability that the character to be generated belongs to each character in the character table in sequence according to the corresponding clause adjustment character order, and then select the character probability corresponding to the adjustment character from the character probabilities that the character to be generated belongs to each character in the character table until the number of character probabilities corresponding to the adjustment characters generated for this adjusted description clause reaches the number of clause adjustment characters, and stop the prediction and generation of character probabilities. Thus, the initial adjusted clause probability corresponding to the adjusted description clause can be obtained. The specific generation method is similar to the method of generating the third-word probability set in the foregoing embodiment, and will not be elaborated here.

[0132] Similarly, it can be known that during the training process, the server can also determine the actual adjusted clause probability of each adjusted description clause from the fourth-word probability set in a similar manner as described above.

[0133] The following details how to determine the model loss. In one specific embodiment, using the first and third word probability sets as references, and based on the second and fourth word probability sets, a second model loss is determined. This includes: determining the model text loss based on the ratio between the first and second word probability sets and the ratio between the third and fourth word probability sets; determining the model sentence loss based on the ratio between the initial and actual descriptive sentence probabilities and the ratio between the initial and actual adjusted sentence probabilities; and finally, determining the model loss as the sum of the model text loss and the model sentence loss.

[0134] Specifically, the server determines the model text loss based on the ratio between the first and second word probability sets, and the ratio between the third and fourth word probability sets. That is, the server performs logarithmic processing on the ratio between the first and second word probability sets, and then multiplies the logarithmic result with the balance parameter to obtain the product result. Similarly, the server performs logarithmic processing on the ratio between the third and fourth word probability sets, and then multiplies the logarithmic result with the balance parameter to obtain the product result. Finally, the model text loss is determined based on the difference between the two product results. For easier understanding of the above processing method, see formula (4):

[0135] (4)

[0136] in, Represents the model's text loss. Represents the probability set of the fourth word. Represents the probability set of the third word. Represents the probability set of the second word. Represents the probability set of the first word. This represents the balance parameter. The balance parameter is typically set to 0.1.

[0137] Furthermore, the model sentence loss is determined based on the ratio between the initial and actual descriptive sentence probabilities, and the ratio between the initial and actual adjusted sentence probabilities. Specifically, for each adjusted descriptive sentence, the server performs logarithmic processing on the ratio between the initial and actual adjusted sentence probabilities, and then multiplies the logarithmic result with the balancing parameter to obtain the first multiplication result. Then, for both the initial and actual descriptive sentences, the server performs logarithmic processing on the ratio between the initial and actual descriptive sentence probabilities, and then multiplies the logarithmic result with the balancing parameter to obtain the second multiplication result. Finally, the model sentence loss is determined based on the difference between the first and second multiplication results. For easier understanding of the above processing method, see formula (5):

[0138] (5)

[0139] in, Indicates adjustment of descriptive clauses The model's sentence segmentation loss, This indicates the actual probability of adjusting the clause. This indicates the probability of actually describing a clause. This indicates the probability of actually describing a clause. Indicates the probability of initial descriptive clause segmentation. This represents the balance parameter. The balance parameter is typically set to 0.1.

[0140] Based on this, the server determines the model loss as the sum of the model text loss and the model sentence loss. That is, the server specifically sums the model text loss and the model sentence loss for each adjusted descriptive sentence, and determines the summation result as the model loss. For easier understanding of the above processing method, see formula (6):

[0141] (6)

[0142] in, Indicates the model loss. Represents the model's text loss. Indicates adjustment of descriptive clauses The model's sentence segmentation loss, This indicates an adjustment to the number of descriptive clauses.

[0143] Step 510: Determine the model loss based on the first model loss and the second model loss.

[0144] Specifically, the model loss is determined based on the first model loss and the second model loss. That is, the server determines the model loss by summing the first model loss and the second model loss. In other words, the first model loss and the second model loss are summed, and the sum is used as the model loss. For ease of understanding, see formula (7):

[0145] (7)

[0146] in, Indicates the model loss. This represents the loss of the first model. This represents the loss of the second model.

[0147] In this embodiment, an open-set detector is used to help determine whether there is noise in the descriptive text that does not match the image. Based on the descriptive text trained in this way, which allows the detector to output with high confidence, the inaccuracy of standards that may be caused by manual adjustment is further considered. Therefore, text adjustment processing is performed on the initial descriptive text, including text correction, text deletion, and text filling. The resulting adjusted descriptive text is determined as the expected output result, so that the model can use the adjusted descriptive text as the training target during optimization. This ensures that parameter tuning is performed based on the initial descriptive text and with the goal of generating adjusted descriptive text. The model loss is calculated by using the two methods mentioned above to complete the optimization process of the model, so as to ensure that the optimized image descriptive text generation model can generate more accurate image descriptive text, that is, improve the accuracy of image descriptive text generation.

[0148] Based on the detailed description of the foregoing embodiments, the complete process of the optimization method for the image description text generation model in the embodiments of this application will be described below. In one embodiment, such as Figure 7 As shown, an optimization method for an image description text generation model is provided, which is then applied to... Figure 1 Taking server 104 as an example for illustration, it can be understood that this method can also be applied to terminal 102 with model training computing power, and can also be applied to a system including terminal 102 and server 104, and implemented through the interaction between terminal 102 and server 104. In this embodiment, the image samples include a first image sample and a second image sample as an example for description, and the method includes the following steps:

[0149] Step 701: Before training the image description text generation model, input the image sample into the image description text generation model to obtain the first word probability set of the character to be generated belonging to each character in the character table, and determine the first word probability set of the character to be generated belonging to each character in the character table.

[0150] Step 702: During the training of the image description text generation model, the image sample is input into the image description text generation model to obtain the actual output probability of the character to be generated belonging to each character in the character table, and the actual output probability of the character to be generated belonging to each character in the character table is determined as the second word probability set.

[0151] Step 703: Determine the probability generation order of each actual output probability in the second word probability set; determine the character that matches the maximum actual output probability of the character to be generated belonging to each character in the word table as the actual generated character; based on the probability generation order, concatenate the actual generated characters in sequence to obtain the actual description text.

[0152] Step 704: Segment the actual description text to obtain multiple actual description sentences.

[0153] Step 705: Perform open set recognition on the image samples and each actual descriptive sentence to obtain the actual sentence confidence of each actual descriptive sentence.

[0154] Step 706: Average the confidence scores of each actual descriptive clause to obtain the average confidence score; calculate the consistency loss based on the first word probability set and the second word probability set; determine the first model loss using the average confidence score and the consistency loss.

[0155] Step 707: Determine the initial description text represented by the first word probability set, and determine the adjusted description text based on the initial description text.

[0156] Step 708: Obtain the probability set of the third word output by the image description text generation model before training begins, with the adjusted description text as the output target, and the probability set of the fourth word output during training, with the adjusted description text as the output target.

[0157] Step 709: Using the first and third word probability sets as references, and based on the second and fourth word probability sets, determine the second model loss.

[0158] Step 710: Determine the model loss based on the first model loss and the second model loss.

[0159] Step 711: Based on the model loss, adjust the model parameters of the image description text generation model to obtain the optimized image description text generation model.

[0160] It should be understood that the specific implementation methods of steps 701 to 711 are similar to those of the aforementioned embodiments, and will not be repeated here.

[0161] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages in other steps.

[0162] Based on the same inventive concept, this application also provides an optimization apparatus for an image description text generation model to implement the optimization method for the image description text generation model described above. The solution provided by this apparatus is similar to the implementation scheme described in the above method. Therefore, the specific limitations of one or more image description text generation model optimization apparatus embodiments provided below can be found in the limitations of the image description text generation model optimization method described above, and will not be repeated here.

[0163] In one embodiment, such as Figure 8 As shown, an optimization device for an image description text generation model is provided, comprising: a probability set acquisition module 802, an open set recognition module 804, a model loss determination module 806, and a model optimization module 808, wherein:

[0164] The probability set acquisition module 802 is used to acquire the first word probability set output by the image description text generation model before training begins and the second word probability set output during training, using image samples as input data.

[0165] The open set recognition module 804 is used to perform open set recognition based on the image samples and the actual descriptive text represented by the second word probability set, and to obtain the text confidence of the actual descriptive text.

[0166] The model loss determination module 806 is used to determine the model loss based on the second word probability set and the text confidence, with the first word probability set as a reference.

[0167] The model optimization module 808 is used to adjust the model parameters of the image description text generation model based on the model loss, so as to obtain an optimized image description text generation model.

[0168] In one embodiment, the probability set acquisition module is specifically used to input image samples into the image description text generation model before training begins, to obtain a first word probability set of the character to be generated belonging to each character in the word table, and to determine the first word probability set of the character to be generated belonging to each character in the word table; during the training process of the image description text generation model, image samples are input into the image description text generation model to obtain the actual output probability of the character to be generated belonging to each character in the word table, and to determine the actual output probability of the character to be generated belonging to each character in the word table as the second word probability set.

[0169] In one embodiment, the probability set acquisition module is specifically used to input image samples into the image description text generation model during the training process of the image description text generation model, and sequentially generate the actual output probability of the character to be generated belonging to each character in the word table that constitutes the actual description text; when the actual output probability of the character to be generated belonging to the end character in the word table is the maximum value, the output of the actual output probability is stopped, so as to obtain the actual output probability of the character to be generated belonging to each character in the word table.

[0170] In one embodiment, the probability set acquisition module is further configured to determine the probability generation order of each actual output probability in the second word probability set;

[0171] In one embodiment, the probability set acquisition module is specifically used to determine the character that matches the maximum value of the actual output probability of the character to be generated belonging to each character in the character table as the actual generated character; and to concatenate the actual generated characters sequentially based on the probability generation order to obtain the actual description text.

[0172] In one embodiment, the model loss determination module is specifically used to calculate the consistency loss based on the first word probability set and the second word probability set; and to determine the model loss by using the text confidence and the consistency loss.

[0173] In one embodiment, the optimization apparatus for the image description text generation model further includes a sentence segmentation module;

[0174] The sentence segmentation module is used to segment the actual descriptive text into sentences, resulting in multiple actual descriptive sentences;

[0175] The open set recognition module is specifically used to perform open set recognition on the image samples and each actual descriptive sentence to obtain the actual sentence confidence of each actual descriptive sentence; the text confidence includes the actual sentence confidence of each actual descriptive sentence.

[0176] In one embodiment, the model loss determination module is specifically used to perform confidence averaging on the actual sentence confidence of each actual descriptive sentence to obtain an average confidence; calculate the consistency loss based on the first word probability set and the second word probability set; and determine the model loss using the average confidence and the consistency loss.

[0177] In one embodiment, the open set identification module is specifically used to determine open set descriptive sentences that include nouns from the actual descriptive sentences; and to perform open set identification on the image samples and each open set descriptive sentence to obtain the actual sentence confidence of each open set descriptive sentence.

[0178] In one embodiment, when the image description text generation model outputs the first word probability set before training begins, the model parameters of the image description text generation model are in a parameter frozen state; when the image description text generation model outputs the second word probability set during training, the model parameters of the image description text generation model are in a parameter adjustable state.

[0179] In one embodiment, the optimization apparatus for the image description text generation model further includes a text determination module;

[0180] The text determination module is used to determine the initial description text represented by the first word probability set, and to determine the adjusted description text based on the initial description text;

[0181] The probability set acquisition module is also used to acquire the probability set of the third word output by the image description text generation model before training begins, with the adjusted description text as the output target, and the probability set of the fourth word output during training with the adjusted description text as the output target.

[0182] The model loss determination module is specifically used to determine the first model loss based on the second word probability set and the text confidence, with the first word probability set as a reference.

[0183] Using the first and third word probability sets as references, and based on the second and fourth word probability sets, determine the second model loss;

[0184] The model loss is determined based on the first model loss and the second model loss.

[0185] In one embodiment, the model loss determination module is specifically used to determine the second model loss based on the ratio between the first word probability set and the second word probability set, and the ratio between the third word probability set and the fourth word probability set.

[0186] In one embodiment, a computer device is provided, which can be a server or a terminal. This embodiment uses a server as an example for description, and its internal structure diagram is as follows: Figure 9 As shown, the computer device includes a processor, memory, input / output (I / O) interfaces, and a communication interface. The processor, memory, and I / O interfaces are connected via a system bus, and the communication interface is also connected to the system bus via the I / O interfaces. The processor provides computational and control capabilities. The memory includes a non-volatile storage medium and internal memory. The non-volatile storage medium stores an operating system, computer programs, and a database. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage medium. The database contains data related to the embodiments of this application, such as image samples, a first word probability set, a second word probability set, a third word probability set, a fourth word probability set, and an image description text generation model. The I / O interfaces are used for exchanging information between the processor and external devices. The communication interface is used for communicating with external terminals via a network connection. When the computer program is executed by the processor, it implements an optimization method for an image description text generation model.

[0187] Those skilled in the art will understand that Figure 9 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0188] In one embodiment, a computer device is also provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps in the above method embodiments.

[0189] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon that, when executed by a processor, implements the steps in the above method embodiments.

[0190] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps in the above method embodiments.

[0191] It should be noted that the object information (including but not limited to object device information, object personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the object or fully authorized by all parties, and the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant countries and regions.

[0192] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments described above. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, etc., and are not limited to these.

[0193] The technical features in the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0194] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. An optimization method for an image-description text generation model, characterized in that, include: Using image samples as input data, obtain the first word probability set output by the image description text generation model before training begins and the second word probability set output during training. Based on the image samples and the actual descriptive text represented by the second word probability set, open set recognition is performed to obtain the text confidence of the actual descriptive text; Using the first word probability set as a reference, the model loss is determined based on the second word probability set and the text confidence. Based on the model loss, the model parameters of the image description text generation model are adjusted to obtain an optimized image description text generation model.

2. The method according to claim 1, characterized in that, The step of using image samples as input data to obtain the first word probability set output by the image description text generation model before training begins and the second word probability set output during training includes: Before training the image description text generation model, the image sample is input into the image description text generation model to obtain the first word probability set of the character to be generated belonging to each character in the character table. The first word probability set of the character to be generated belonging to each character in the character table is determined as the first word probability set. During the training of the image description text generation model, the image sample is input into the image description text generation model to obtain the actual output probability of the character to be generated belonging to each character in the character table, and the actual output probability of the character to be generated belonging to each character in the character table is determined as the second word probability set.

3. The method according to claim 2, characterized in that, During the training process of the image description text generation model, the image sample is input into the image description text generation model to obtain the actual output probability of the character to be generated belonging to each character in the character table, including: During the training process of the image description text generation model, the image sample is input into the image description text generation model, and the actual output probability of the characters to be generated that constitute the actual description text belonging to each character in the character table is generated in sequence. When the actual output probability of the character to be generated belonging to the end character in the word table is at its maximum value, the output of the actual output probability is stopped to obtain the actual output probability of the character to be generated belonging to each character in the word table.

4. The method according to claim 2, characterized in that, The method further includes: Determine the probability generation order of each actual output probability in the second word probability set; The character that matches the maximum value of the actual output probability of the character to be generated belonging to each character in the character table is determined as the actual generated character; Based on the probability generation order, the actual generated characters are concatenated sequentially to obtain the actual description text.

5. The method according to claim 3, characterized in that, The step of determining the model loss based on the first word probability set, the second word probability set, and the text confidence score, includes: Calculate the consistency loss based on the first word probability set and the second word probability set; The model loss is determined by the text confidence score and the consistency loss.

6. The method according to claim 1, characterized in that, The method further includes: The actual description text is segmented into sentences to obtain multiple actual description sentences; The step of performing open-set recognition based on the image samples and the actual descriptive text represented by the second word probability set to obtain the text confidence of the actual descriptive text includes: Open set recognition is performed on the image samples and each actual descriptive sentence to obtain the actual sentence confidence of each actual descriptive sentence; the text confidence includes the actual sentence confidence of each actual descriptive sentence.

7. The method according to claim 6, characterized in that, The step of determining the model loss based on the first word probability set, the second word probability set, and the text confidence score, includes: The confidence scores of each actual descriptive clause are averaged to obtain the average confidence score. Calculate the consistency loss based on the first word probability set and the second word probability set; The model loss is determined by the average confidence level and the consistency loss.

8. The method according to claim 6, characterized in that, The step of performing open-set recognition on the image samples and each actual descriptive sentence to obtain the actual sentence confidence score for each actual descriptive sentence includes: Identify open set descriptive clauses that include nouns from the actual descriptive clauses; Open set identification is performed on the image samples and each of the open set descriptive sentences to obtain the actual sentence confidence of each of the open set descriptive sentences.

9. The method according to claim 1, characterized in that, When the image description text generation model outputs the first word probability set before training begins, the model parameters of the image description text generation model are in a parameter frozen state. When the image description text generation model outputs the second word probability set during the training process, the model parameters of the image description text generation model are in an adjustable state.

10. The method according to claim 3, characterized in that, The method further includes: Determine the initial description text represented by the first word probability set, and determine the adjusted description text based on the initial description text; Obtain the probability set of the third word output by the image description text generation model before training begins, with the adjusted description text as the output target, and the probability set of the fourth word output during training with the adjusted description text as the output target; The step of determining the model loss based on the first word probability set, the second word probability set, and the text confidence score, includes: Using the first word probability set as a reference, and based on the second word probability set and the text confidence, determine the first model loss; Using the first word probability set and the third word probability set as references, and based on the second word probability set and the fourth word probability set, determine the second model loss; The model loss is determined based on the first model loss and the second model loss.

11. The method according to claim 10, characterized in that, The step of determining the second model loss based on the first word probability set and the third word probability set, and on the second word probability set and the fourth word probability set, includes: The second model loss is determined based on the ratio between the first word probability set and the second word probability set, and the ratio between the third word probability set and the fourth word probability set.

12. An optimization device for an image-description text generation model, characterized in that, The device includes: The probability set acquisition module is used to obtain the first word probability set output by the image description text generation model before training begins and the second word probability set output during training, using image samples as input data. The open set recognition module is used to perform open set recognition based on the image sample and the actual descriptive text represented by the second word probability set, and obtain the text confidence of the actual descriptive text; The model loss determination module is used to determine the model loss based on the second word probability set and the text confidence, with the first word probability set as a reference. The model optimization module is used to adjust the model parameters of the image description text generation model based on the model loss, so as to obtain an optimized image description text generation model.

13. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 12.

14. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 12.

15. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 12.

Images