Sample set construction method, question and answer model training method, question and answer processing method, request processing method, task platform

Abstract

The embodiments of this specification provide a method for constructing a sample set, a method for training a question-and-answer model, a method for question-and-answer processing, a method for request processing, and a task platform. Among them, the method for constructing a sample set includes: inputting a sample image into an image annotation model to obtain a global annotation result of the sample image; identifying a target object in the sample image, and based on the obtained identification result, marking the target object in the sample image to obtain a global view and intercepting the target object to obtain a local view; inputting the global view, the local view, and visual prompt information into the image annotation model to obtain a local annotation result of the sample image, where the visual prompt information is used to prompt the image annotation model to perform an annotation task on the global view based on the local view; constructing a sample set based on the sample image, the global annotation result, and the local annotation result. This method can improve the accuracy and comprehensiveness of the annotation result for an image.

Description

Technical Field

[0001] The embodiments of this specification relate to the field of computer technology, and particularly to a method for constructing a sample set, a method for training a question-and-answer model, a method for processing questions and answers, a method for processing requests, and a task platform. Background Art

[0002] With the development of computer technology and artificial intelligence technology, machine learning models have gradually been widely applied in various scenarios.

[0003] If a machine learning model is to be used in a certain scenario, it is necessary to use the training data related to this scenario for model training, or, for a pre-trained machine learning model, fine-tune the pre-trained machine learning model based on the training data related to this scenario to obtain a machine learning model that can be put into use. The sample data used for model training includes the original data and its labels, and the quality of the sample data plays a crucial role in the training effect of the model. Usually, it is necessary to manually annotate the original data, that is, to label the original data, to obtain the sample data. For example, for an image analysis model, it is necessary to annotate the sample images, such as annotating the description information of the objects included in the sample images.

[0004] Currently, the annotation effect of data depends on the professionalism of the annotators, and it is difficult to ensure the accuracy and comprehensiveness of the annotation results. Correspondingly, it will lead to insufficient performance of the trained model. Summary of the Invention

[0005] The embodiments of this specification provide a method for constructing a sample set, which can improve the accuracy and comprehensiveness of the annotation results for images, and correspondingly improve the performance of the model trained based on the sample set. One or more embodiments of this specification simultaneously relate to a method for training a question-and-answer model, a method for processing questions and answers, a method for processing requests based on the question-and-answer model, a task platform, a computing device, an electronic device, a computer-readable storage medium, and a computer program product.

[0006] Among them, the method for constructing a sample set includes:

[0007] Input the sample image into an image annotation model to obtain the global annotation result of the sample image;

[0008] Identify the target object in the sample image, and based on the obtained recognition result, mark the target object in the sample image to obtain a global view and intercept the target object to obtain a local view;

[0009] Input the global view, the local view, and the visual hint information into the image annotation model to obtain the local annotation result of the sample image, where the visual hint information is used to prompt the image annotation model to perform an annotation task on the global view based on the local view;

[0010] Construct a sample set based on the sample image, the global annotation result, and the local annotation result.

[0011] In one embodiment of this specification, on the basis of annotating the global view of a sample image, object recognition is also performed on the sample image and the corresponding local view is intercepted. For the global view and the local view, visual hint information is used to instruct the image annotation model to perform local annotation on the image, and a sample set is constructed based on the global annotation result and the local annotation result. In this way, more comprehensive and complete annotation of the sample image can be ensured, and the accuracy and comprehensiveness of the annotation result can be improved. Correspondingly, the sample set obtained based on this annotation result can achieve a better training effect on the model and ensure a better-performing model. Description of the Drawings

[0012] Figure 1 is a schematic structural diagram of a question-and-answer system provided by an embodiment of this specification;

[0013] Figure 2 is a flowchart of a method for constructing a sample set provided by an embodiment of this specification;

[0014] Figure 3 is an overall implementation framework diagram of a method for constructing a sample set provided by an embodiment of this specification;

[0015] Figure 4 is a flowchart of a method for training a question-and-answer model provided by an embodiment of this specification;

[0016] Figure 5 is a flowchart of a method for processing questions and answers provided by an embodiment of this specification;

[0017] Figure 6 is a flowchart of a method for processing requests based on a question-and-answer model provided by an embodiment of this specification;

[0018] Figure 7 is a schematic structural diagram of a task platform provided by an embodiment of this specification;

[0019] Figure 8 is a schematic structural diagram of a device for constructing a sample set provided by an embodiment of this specification;

[0020] Figure 9 is a schematic structural diagram of a device for training a question-and-answer model provided by an embodiment of this specification;

[0021] Figure 10 It is a schematic structural diagram of a question-and-answer processing device provided by an embodiment of this specification;

[0022] Figure 11 It is a structural block diagram of a computing device provided by an embodiment of this specification;

[0023] Figure 12 It is a structural block diagram of an electronic device provided by an embodiment of this specification. Specific embodiments

[0024] In the following description, many specific details are set forth in order to provide a thorough understanding of this specification. However, this specification can be implemented in many other ways different from those described herein, and those skilled in the art can make similar generalizations without departing from the connotation of this specification. Therefore, this specification is not limited by the specific embodiments disclosed below.

[0025] The terms used in one or more embodiments of this specification are for the purpose of describing specific embodiments only and are not intended to limit one or more embodiments of this specification. The singular forms "a" and "the" used in one or more embodiments of this specification and the appended claims are also intended to include the plural forms unless the context clearly dictates otherwise. It should also be understood that the term "and / or" used in one or more embodiments of this specification refers to and includes any or all possible combinations of one or more of the associated listed items.

[0026] It should be understood that although the terms first, second, etc. may be used in one or more embodiments of this specification to describe various information, such information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of one or more embodiments of this specification, the first may also be referred to as the second, and similarly, the second may also be referred to as the first. Depending on the context, the word "if" as used herein can be interpreted as "when" or "while" or "in response to determining".

[0027] In addition, it should be noted that the user information (including but not limited to user device information, user personal information, user behavior information, etc.) and data (including but not limited to data for analysis, stored data, displayed data, etc.) involved in one or more embodiments of this specification are all information and data that have been authorized by the user or fully authorized by all parties, and the collection, use, and processing of the relevant data need to comply with the relevant laws, regulations, and standards of the relevant countries and regions, and corresponding operation entrances are provided for the user to select authorization or rejection.

[0028] In one or more embodiments of this specification, a large model refers to a deep learning model with a large number of model parameters, usually including hundreds of millions, tens of billions, hundreds of billions, trillions or even more than one quadrillion model parameters. A large model can also be called a Foundation Model. Through pre-training of the large model with a large amount of unlabeled corpus, a pre-trained model with more than hundreds of millions of parameters is produced. This kind of model can adapt to a wide range of downstream tasks and has good generalization ability. For example, large language models (LLMs), multi-modal pre-training models, etc.

[0029] When a large model is actually applied, only a small number of samples are needed to fine-tune the pre-trained model for application to different tasks. Large models can be widely applied in the fields of natural language processing (NLP), computer vision, etc. Specifically, they can be applied to tasks in the field of computer vision such as visual question answering (VQA), image captioning (IC), image generation, etc., and tasks in the field of natural language processing such as text-based sentiment classification, text data annotation, machine translation, etc. The main application scenarios of large models include digital assistants, intelligent robots, search, online education, office software, e-commerce, intelligent design, etc. The models involved in the embodiments of this specification include, but are not limited to, the above-mentioned large models, and can be any type of machine learning model. For example, it can be a BERT model, a convolutional neural network (CNN) model, a Transformer model, etc.

[0030] The models involved in the embodiments of this specification can adopt large models or other models with smaller scales, which are not limited in the embodiments of this specification.

[0031] With the development of computer technology and artificial intelligence technology, machine learning models are gradually used to assist in performing tasks in various fields, and the requirements for the performance of machine learning models are getting higher and higher. The performance of a machine learning model depends to a great extent on the quality of the sample data used for its training. If high-quality sample data is used to train a machine learning model, the performance of the trained model will be higher. For models used in different fields, if data related to that field is used for model training, the application effect of the trained model in that field will also be better. Therefore, the construction effect of the sample data required for model training is crucial.

[0032] In recent years, large models have gradually evolved from single-modality to general AI that integrates multiple modalities. Companies have launched a surge in large models, demonstrating strong performance in general-purpose tasks such as visual question answering (VQA) and image captioning. However, in industries requiring more specialized knowledge, the application of large models still faces numerous practical challenges. This often requires building richer, more specialized sample sets (such as industry image and text datasets) for each large model, allowing for fine-tuning of the model to enhance its specialized capabilities.

[0033] The annotation and dataset construction methods for graphic and text data are still in a relatively primitive state from the era of traditional visual models. That is, it is necessary to manually frame the target object in the image or manually describe the content in the image, which consumes a lot of annotation labor costs. At the same time, for more professional industry fields, the annotation results are very dependent on the professionalism of the annotators. The quality of the annotation results provided by annotators with lower professionalism may not meet the requirements. Annotators usually perform annotations based on given fixed templates, which will lead to low data quality problems such as severe templateization of annotation results and high content error rate. For images containing many entities, it is usually difficult for annotators to provide an accurate and comprehensive description of the image and text. For the labeled data, it is usually difficult to obtain the required model capabilities by directly using it for training. Data processing links such as data cleaning, industry content supplementation, and data evolution enhancement are also required.

[0034] The embodiments of this specification provide a sample set construction method that can accurately and comprehensively annotate sample images, construct a high-quality sample set, and correspondingly improve the performance of a model trained based on the sample set. One or more embodiments of this specification also relate to a question-answering model training method, a question-answering processing method, a request processing method based on a question-answering model, a task platform, a computing device, an electronic device, a computer-readable storage medium, and a computer program product.

[0035] In the embodiments of this specification, the sample set constructed using the sample set construction method can be used to train a question-answering model, and the question-answering model is used to implement the question-answering processing method. Figure 1 This is a schematic diagram of the structure of a question-answering system provided in the embodiment of this specification. The question-answering processing method provided in the embodiment of this specification can be applied to the question-answering system. Figure 1 As shown, the question-answering system may include a cloud platform 10 and a terminal device 20 that are communicatively connected.

[0036] The terminal device 20 is used to: send problem information to be solved to the cloud platform 10. The cloud platform 10 is used to: execute the question-and-answer processing method provided in the embodiments of this specification based on the problem information, obtain the reply information corresponding to the problem information, and send the reply information to the terminal device 20. The cloud platform 10 can be the background server of some applications or web pages with question-and-answer functions, and is used to support the normal operation of the application. The application can be installed in the terminal device 20. By running the application, the terminal device 20 realizes the communication connection with the cloud platform 10 and interacts with the cloud platform 10 to complete the question-and-answer task. In some embodiments, in addition to providing the question-and-answer function, the application can also provide other functions, such as social communication function, video playback function, image processing function, etc.

[0037] In an application scenario as Figure 1 shown, the question-and-answer model (such as a large model) used in the implementation process of the question-and-answer processing method is deployed in the cloud platform 10. The cloud platform 10 can be connected to one or more terminal devices 20 through a local area network connection, a wide area network connection, an Internet connection, or other types of data networks. Here, the terminal devices 20 can include but are not limited to: smart phones, tablet computers, laptop computers, handheld computers, personal computers, smart home devices, vehicle-mounted devices, etc. The terminal device 20 can interact with the user through a graphical user interface to realize the invocation of the question-and-answer model, and further realize the question-and-answer processing method provided in the embodiments of this specification. If the computing resources of the terminal device 20 are large enough, the large model can also be directly deployed on the terminal device, and the terminal device directly uses the large model to realize the question-and-answer processing method provided in the embodiments of this specification. The question-and-answer model can also be trained on the cloud platform 10. Correspondingly, the training method of the question-and-answer model provided in this specification can also be executed by the cloud platform 10.

[0038] The sample set construction method provided in the embodiments of this specification can also be executed by the cloud platform 10. The image annotation model used in the sample set construction method can be deployed in the cloud platform 10. In some embodiments, the sample set construction method can also be executed by the terminal device 20. For example, the terminal device 20 invokes the image annotation model deployed on the cloud platform 10 to execute the sample set construction method, or the image annotation model can be deployed in the terminal device 20.

[0039] There is also a certain feedback mechanism between the terminal device 20 and the cloud platform 10. The terminal device 20 can also send feedback information about the reply information to the cloud platform 10, and the cloud platform 10 can adjust the question-and-answer model based on the feedback information. The cloud platform 10 can also update the sample set used in the training process of the question-and-answer model based on the feedback information, so that the terminal device 20 continues to train and adjust the question-and-answer model based on the updated sample set.

[0040] Figure 2 is a flowchart of a sample set construction method provided by an embodiment of this specification. Below, an example will be given where the sample set construction method is executed by a construction device. In some embodiments, this method can be applied to Figure 1 the cloud platform 10 in, for example, the construction device can be the cloud platform 10. As Figure 2 shown, this method can include the following steps 202 to step 208.

[0041] Step 202: Input the sample image into the image annotation model to obtain the global annotation result of the sample image.

[0042] The image annotation model in the embodiments of this specification can be an existing trained image annotation model. This image annotation model can be a vision-language model (VLM), which belongs to an artificial intelligence model that combines visual information and language information.

[0043] The sample image can be an image collected by a staff member, or an image collected from the Internet using a computing device, or an image provided by a customer, which is not limited here. In some embodiments, the sample image can be related to the application field and scenario of the model to be trained. For example, the constructed sample set is used to train the question-and-answer scenario in the natural scenery field. Correspondingly, the sample images used to construct the sample set can more preferably be natural scenery images. Another example is that the constructed sample set is used to train the question-and-answer scenario in the field of celestial bodies and astronomical phenomena. Correspondingly, the sample images used to construct the sample set can more preferably be space celestial body images.

[0044] The construction device can input the sample image into the image annotation model, so that the image annotation model analyzes and annotates the sample image according to its originally set annotation logic to obtain the global annotation result. This global annotation result can be the description information for this sample image. In one example, this description information includes information such as the general content included in the sample image, the field to which the sample image belongs, and the overall style of the sample image.

[0045] In some embodiments, the model to which the constructed sample set is to be applied can be determined in advance, and the task requirement information for this model is obtained. This task requirement information indicates the requirements for the model, such as indicating the field and scenario to which the model is applied, the tasks that the model needs to execute, and can also indicate the task execution logic of the model. The construction device can perform corresponding annotation on the sample image based on this task requirement information to ensure that the obtained annotation result matches the model to be trained and ensure a better model training result based on this annotation result.

[0046] To call an image annotation model to perform an annotation task, it is necessary to input a prompt to the image annotation model to prompt the model to perform the corresponding processing process and output a result that meets the requirements. A prompt refers to a prompt word, which is a type of natural language input, similar to a command or instruction, used to indicate what actions the model should take or what output it should generate when performing a specific task. In the embodiments of this specification, the corresponding prompt can be constructed based on the task requirement information to instruct the image annotation model to output a global annotation result that meets the requirements.

[0047] Correspondingly, before the above step 202, the sample set construction method provided in the embodiments of this specification further includes: constructing a preset prompt based on the task requirement information; wherein, the task requirement information indicates the target task to be performed by the model to be trained; the preset prompt is used to prompt the image annotation model to extract the information of the input image according to the target content type corresponding to the target task for global annotation. In step 202, inputting the sample image into the image annotation model to obtain the global annotation result of the sample image includes: inputting the sample image and the preset prompt into the image annotation model, so that the image annotation model extracts the information of the sample image according to the target content type corresponding to the target task, and obtains the global annotation result of the sample image; wherein, the global annotation result includes the information of the target content type.

[0048] For different models, such as models applied to different scenarios or implementing different functions, the corresponding task requirement information can be different. The task requirement information at least indicates the target task to be performed by the model. A model can perform one or more target tasks, and the number of target tasks indicated by the task requirement information can be one or more. By way of example, the target tasks that a question-and-answer model can perform on an image can include: providing a text description of the image, answering questions about the image, or determining the corresponding image attributes from a given set of options for the image. The target tasks can also include other tasks, which are not limited herein.

[0049] For different target tasks, different types of content can be annotated for the image to train the model based on the obtained annotation content to achieve the target task. Each target task corresponds to a target content type to be annotated. By way of example, for the target task of providing a text description of an image, the corresponding target content type is the text description information of the image. For the target task of answering questions about an image, the corresponding target content type is the reply information to the given sample questions determined for the image. For the target task of determining the image attributes of an image, the corresponding target content type is the image attributes to which the image belongs.

[0050] The construction device constructs preset prompt information based on the target content types corresponding to at least one target task indicated by the task requirement information. The preset prompt information is used in the global annotation process of the input sample image to prompt the image annotation model to analyze the sample image and extract information, obtain the information of the target content type, and use the information of the target content type as the global annotation result of the sample image. The construction device inputs the sample image and the preset prompt information into the image annotation model together, so that the image annotation model processes based on the prompt of the preset prompt information.

[0051] In some embodiments, for the model to be trained, the construction device can construct the preset prompt information only once based on the task requirement information corresponding to the model, and the preset prompt information can be applicable to all sample images. The same preset prompt information can be used for each sample image to obtain the global annotation result of the sample image, without separately constructing new prompt information for different sample images.

[0052] In the embodiments of this specification, the preset prompt information is used to instruct the image annotation model to implement the global annotation of the image. The preset prompt information is constructed based on the task requirement information of the model to be trained. In this way, it can be ensured that the obtained annotation result can better match the model, the obtained annotation result is more suitable for model training, and a model with better performance can be obtained through training.

[0053] Step 204: Identify the target object in the sample image, and based on the obtained identification result, mark the target object in the sample image to obtain a global view and intercept the target object to obtain a local view.

[0054] In the embodiments of this specification, in addition to performing global annotation on the sample image, the construction device also performs local annotation on the sample image to improve the integrity and comprehensiveness of the annotation of the sample image. For local annotation, the construction device first needs to determine the local information in the sample image (i.e., the information located in the local area), and then analyze the local information to implement local annotation. The local annotation can be performed on certain specific objects in the sample image. By way of example, the construction device can perform object recognition on the sample image to identify the target object in the sample image, and obtain the local information where the target object is located in the sample image based on the target object. The local information includes information about the target object, such as the category to which the target object belongs, the current state of the target object, and the relationship between the target object and surrounding objects. The target object can also be referred to as a detailed entity in the image.

[0055] In one implementation, the local information in the sample image includes the local view in the sample image, such as the local view where the target object is located. The construction device can use techniques such as image detection and segmentation, such as using vision models for image detection and segmentation to determine the local view in the sample image that contains the target object, and perform local annotation based on this local view. For example, the construction device can intercept the target object recognized in the sample image to obtain the local view. During the process of local annotation, the construction device can also combine the global view of the sample image so that the obtained local annotation result includes the association information between the information in this local view and the information in the global view, ensuring more complete and detailed information is obtained and ensuring a relatively high integrity of the annotation of the sample image. For example, this global view can be a view obtained by marking the target object on the basis of the sample image.

[0056] For different models, there are differences in the categories of target objects that need to be determined. By way of example, for an image that contains both a person and a plant, the human behavior analysis model focuses on the person. If the human behavior analysis model is trained, then the target object determined during annotation should be the person, and thus information about the person is obtained; the plant recognition model focuses on the plant. If the plant recognition model is trained, then the target object determined during annotation should be the plant, and thus information about the plant is obtained. In some embodiments, the construction device can determine the required object type based on the task requirement information, and then perform object recognition on the sample image based on this object type to determine the target object of this object type. In this way, it is ensured that the local annotation based on the target object matches the model requirements, and the training effect of the model based on the local annotation result is better. The object type determined based on the task requirement information can be one or multiple, and there is no limitation here.

[0057] The construction device can identify the target objects in the sample image, and the identification result can characterize the area where each target object is located. Exemplarily, the identification result can include the detection boxes or surrounding polygons of each target object, and the detection boxes and surrounding polygons indicate the coordinates of the smallest area where the target object is located. The construction device can directly intercept the detection box or the surrounding polygon to obtain a local view. In some embodiments, based on the smallest area where the target object is located, the construction device can adjust the area to intercept a larger area containing the target object as the local view. In this way, more detailed local information can be obtained based on the local view. For example, in addition to obtaining the information of the target object itself, the association relationship between the target object and its background can also be obtained, ensuring that the local annotation result is more complete and detailed. Exemplarily, based on the smallest area, the construction device can expand the area range based on a set expansion parameter to obtain a target area containing the target object, and then intercept the target area in the sample image to obtain a local view. The expansion parameter is greater than 1. For example, the expansion parameter can be 1.2, 1.1 or other values.

[0058] In one implementation, if the identified object for the target object is a surrounding polygon, the construction device can construct an outer circumscribed rectangle of the polygon based on the point with the largest coordinate value and the point with the smallest coordinate value in the surrounding polygon. For example, taking the lower left vertex of the sample image as the origin to construct a rectangular coordinate system, the leftmost and lowest point and the rightmost and uppermost point in the surrounding polygon can be used as the two diagonal points of the rectangle to construct the outer circumscribed rectangle. For the outer circumscribed rectangle or the detection box of the target object, calculate the coordinates of the target area by magnifying it by a fixed multiple (such as the above expansion parameter) with its center as the center. For example, the coordinates of the detection box are represented as B = (x, y, w, h), where x and y respectively represent the horizontal and vertical coordinates of the upper left point of the detection box, w represents the width of the detection box, and h represents the height of the detection box. The coordinates of the target area can be B' = scale(B, c) = (x + w / 2 - c×w / 2, y + h / 2 - c×h / 2, c×w, c×h), where c represents the expansion parameter. Then, the target area can be intercepted from the sample image based on the coordinates of the target area to obtain a local view. The construction device can also draw a marked box (such as a red rectangular box) at the corresponding position on the sample image based on the coordinates of the target area and save it as the global view of the sample image.

[0059] In some embodiments, if multiple target objects are identified in the sample image, the corresponding global view and local view can be determined respectively for each target object.

[0060] Step 206: Input the global view, the local view, and the visual prompt information into the image annotation model to obtain the local annotation result of the sample image, where the visual prompt information is used to prompt the image annotation model to perform the annotation task on the global view based on the local view.

[0061] The construction device can continue to use the image annotation model to perform local annotation on the sample image. During the local annotation process, the global view and the local view of the sample image determined for the target object can be combined. For the relevant introduction of the image annotation model, reference can be made to the relevant introduction in the foregoing step 202, which will not be elaborated here.

[0062] The construction device can first construct visual prompt information to instruct the image annotation model to perform the annotation task on the global view based on the local view, so as to realize the local annotation of the sample image. Correspondingly, before inputting the global view, the local view, and the visual prompt information (visual prompt) into the image annotation model in step 206, the sample set construction method provided in the embodiments of this specification further includes: constructing visual prompt information based on the task requirement information; where the task requirement information indicates the target task to be performed by the model to be trained; the visual prompt information is used to prompt the image annotation model to perform the annotation task, extract information from the input local view according to the target content type corresponding to the target task, and determine the association information between the input local view and the global view. The visual prompt information includes a multi-image input instruction, indicating the analysis of the input global view and local view.

[0063] In the embodiments of this specification, the construction device can construct the visual prompt information based on the task requirement information. For the task requirement information, the target task, and the corresponding target content type, reference can be made to the relevant introduction in the foregoing step 202, which will not be elaborated here. The visual prompt information can instruct the image annotation model to extract information from the input local view. For example, extract the information of the target content type in the local view, such as including the information related to the target object therein, and can also include the association information between the target object in the local view and other contents outside it (such as the background of the target object, the scene where it is located, etc.). The visual prompt information also instructs the image annotation model to determine the association information between the input local view and the global view, so as to describe the local view based on the association information and the information extracted from the local view (such as including the information of the target object itself). For example, if the sample image is an image of a playground and the target object in the local view is a child, the obtained local annotation result can be that the sample image contains a child playing in the playground, and the child also interacts with the puppy beside it.

[0064] In the embodiments of this specification, during the process of local annotation by the visual prompt information prompt image annotation model, the association information between the local view and the global view is annotated. In this way, it can be ensured that the local information in the image can be more comprehensively described through local annotation, and the effect of model training based on the local annotation results can be improved.

[0065] Step 208: Construct a sample set based on the sample image, the global annotation result, and the local annotation result.

[0066] For each sample image, the construction device can determine a sample data. The above-mentioned processes of global annotation and local annotation are performed for multiple sample images to obtain multiple sample data. Furthermore, these multiple sample data can form a sample set. Each sample data includes a sample image, as well as the global annotation result and the local annotation result of this sample image. For a sample image, if multiple target objects are detected for it, then for each target object, a local annotation result can be obtained. All the local annotation results of this sample image can belong to one sample data.

[0067] In one implementation, after the construction device obtains the global annotation result and the local annotation result for a sample image, it can directly determine this sample image, the global annotation result, and the local annotation result as one sample data. In another implementation, the construction device can further analyze or process the obtained global annotation result and local annotation result, and construct a sample set based on the data obtained after analysis or processing and the original sample image.

[0068] In the embodiments of this specification, after annotating the sample image, the annotation result can be verified. After the verification passes, the sample set is constructed based on the obtained annotation result to ensure the accuracy and perfection of the annotation result. Correspondingly, step 208 above can include: when both the global annotation result and the local annotation result of the sample image pass the verification, construct a sample set based on the sample image, the global annotation result, and the local annotation result. In the case where the verification fails, re-annotation can be performed on the sample image to construct a sample set based on the re-annotation result. Exemplarily, this verification process can be manually verified using an annotation tool. The verifier views the global annotation result and the local annotation result, checks the generated annotation result in the annotation tool, determines whether the annotation result is accurate and perfect, and modifies the unannotated content or incorrect content. In the embodiments of this specification, the annotation process before verification can be called the pre-annotation process, and the obtained annotation result can be called the pre-annotation result. After verifying the pre-annotation result, the final annotation result can be determined.

[0069] In one implementation, step 208 of constructing a sample set based on the sample image, the global annotation result, and the local annotation result includes:

[0070] Send the sample image, global annotation result, and local annotation result to the front end, and receive the annotation verification information for the sample image sent by the front end;

[0071] When the annotation verification information indicates that the local annotation result fails the verification and carries the region information of the area where the target object is located, intercept the auxiliary area indicated by the region information of the sample image to obtain an updated local view;

[0072] Input the global view, updated local view, and visual prompt information into the image annotation model to obtain an additional local annotation result for the sample image; among them, the visual prompt information is used to prompt the image annotation model to perform an annotation task on the global view based on the updated local view;

[0073] Construct a sample set based on the sample image, global annotation result, and additional local annotation result.

[0074] In the embodiments of this specification, after the construction device generates the corresponding global annotation result and local annotation result for the sample image, these annotation results can be sent to the front end for verification by the verification personnel. The front end refers to the terminal device used by the verification personnel. The terminal device uses the annotation tool installed therein to display the received sample image, global annotation result, and local annotation result for the verification personnel to verify. When it is determined that both the global annotation result and the local annotation result are accurate and complete, the verification personnel can trigger the terminal device to send annotation verification information to the construction device, and this annotation verification information indicates that the annotation verification for the sample image passes. Exemplarily, a confirmation-pass control can be displayed on the page for verifying the annotation result on the terminal device, and the verification personnel can click this control to trigger the corresponding annotation verification information. Based on this annotation verification information, the construction device can generate a sample data based on the sample image, its global annotation result, and local annotation result, and store it in the sample set.

[0075] When the verification personnel determine that there are errors or incomplete markings in the global marking result or the local marking result, the marking content can be modified and supplemented through the terminal device. Controls for modifying the marking content and adding marking content can be displayed on the page of the terminal device for verifying the marking result. The verification personnel can click on the corresponding controls and perform operations on the sample image on the page (such as clicking, drawing a frame, or polygon selection, etc.) to modify or add the marking content. After the verification personnel's operation, it can trigger the generation of marking verification information on the terminal device and send it to the construction device. For example, this marking verification information can indicate that the marking result fails the verification. In some embodiments, the verification personnel can also specifically select to modify and supplement the global marking result or modify and supplement the local marking result. Correspondingly, the generated marking verification information indicates that the global or local marking result fails the verification.

[0076] In the case where the local marking result fails the verification, it may be that the recognition of the target object is incomplete or incorrect. The verification personnel can mark points or draw a rectangular frame for the target object in the sample image to input the region information of the region where the target object is located. Correspondingly, the marking verification information carries the region information of the region where the target object is located. The construction device can intercept the auxiliary region indicated by this region information in the sample image based on this region information (such as calling a segmentation model to obtain the polygon segmentation result of the target object), obtain an updated local view, and then re-execute the local marking process in step 206 above based on this updated local view to obtain an updated local marking result. Then, a sample set can be constructed based on this updated local marking result. For example, the sample image, its global marking result, and the updated local marking result can be determined as a sample data. For this updated local marking result, it can also be continuously verified by the verification personnel and then used to construct the sample set after passing the verification.

[0077] If the verification personnel determine that a certain local marking result is incorrect, then this local marking result can be deleted. For example, make the marking verification information carry a deletion instruction for this local marking result. Correspondingly, the construction device can delete this local marking result based on this marking verification information. Through such deletion and the aforementioned method of determining the updated local marking result, the correction and improvement of the local marking result can be achieved.

[0078] In one way, there may be multiple target objects in a sample image. By performing local annotation on the sample image, multiple local annotation results corresponding to the multiple target objects can be obtained. During the verification process, the multiple local annotation results can be verified separately. The annotation verification information can indicate that the local annotation result corresponding to any target object fails the verification and carry the regional information for that target object. Accordingly, the obtained updated local annotation result is also for that target object. At this time, the original local annotation result of that target object can be directly replaced with the updated local annotation result without changing the original local annotation results of other target objects.

[0079] Optionally, the target object targeted by the annotation verification information can also be an object that has not been locally annotated originally. At this time, based on the original local annotation results, the obtained updated local annotation result can be added to obtain the final local annotation result for the sample image.

[0080] In some embodiments, at least one annotation assistance plugin can be installed in the terminal device to provide different annotation functions, assist the verification personnel in verification, and improve the manual efficiency. The determination of the area where the target object is located can be achieved based on the interactive target segmentation tool, which is an annotation assistance plugin. For example, this plugin can be installed in the terminal device, and the verification personnel can start this plugin during the process of verifying the annotation result.

[0081] In one embodiment, in step 208, constructing a sample set based on the sample image, the global annotation result, and the local annotation result includes:

[0082] Sending the sample image, the global annotation result, and the local annotation result to the front end, and receiving the annotation verification information for the sample image sent by the front end;

[0083] In the case where the annotation verification information indicates that the global annotation result fails the verification and carries the first prompt information, inputting the sample image and the first prompt information into an image annotation model to obtain an updated global annotation result for the sample image; wherein, the first prompt information is used to prompt the image annotation model to extract the information of the input image according to the target content type for global annotation;

[0084] In the case where the annotation verification information indicates that the local annotation result fails the verification and carries the second prompt information, inputting the global view, the local view, and the second prompt information into an image annotation model to obtain an updated local annotation result for the sample image; wherein, the second prompt information is used to prompt the image annotation model to perform an annotation task on the global view based on the local view, and the second prompt information is different from the visual prompt information;

[0085] Constructing a sample set based on the sample image, the updated global annotation result, and the updated local annotation result.

[0086] Regarding the annotation verification information, please refer to the relevant introduction in the previous embodiment above, and details are not described here.

[0087] In the embodiments of this specification, when the global annotation result or the local annotation result fails the verification, it is possible that the hint information used in the process of performing the corresponding annotation by the image annotation model is not accurate enough. The verification personnel can construct more accurate hint information by themselves, so as to use this hint information to instruct the image annotation model to re-perform the global annotation or the local annotation, and obtain more accurate global annotation results and local annotation results. The hint information reconstructed by the verification personnel can be carried in the standard verification information and sent to the construction device. The construction device re-enters this hint information into the image annotation model to instruct the image annotation model to re-perform step 202 or step 206 above, so as to re-perform the global annotation or the local annotation, obtain updated annotation results, and then construct a sample set based on the updated annotation results. For the updated annotation results, the verification personnel can also continue to verify, and then use them to construct a sample set after passing the verification.

[0088] In the embodiments of this specification, the hint information reconstructed for the global annotation task is called the first hint information. This first hint information can be analogous to the foregoing preset hint information, and the first hint information is also used to prompt the image annotation model to extract the information of the input image according to the target content type corresponding to the target task for global annotation. The first hint information is different from the foregoing preset hint information, such as different specific semantics or different expression forms. The first hint information can be obtained by the verification personnel modifying the preset hint information, or it can also be regenerated. When the annotation verification information carries the first hint information, the sample image and the first hint information are input into the image annotation model to obtain the updated global annotation result of the sample image.

[0089] The visual hint information reconstructed for the local annotation task is called the second hint information. This second hint information can be analogous to the foregoing visual hint information, and the second hint information is also used to prompt the image annotation model to perform the annotation task, and extract the information of the local view and its association information with the global view according to the target content type corresponding to the target task. The specific content of the second hint information is different from that of the visual hint information, such as different specific semantics or different expression forms. The second hint information can be obtained by the verification personnel modifying the foregoing visual hint information, or it can also be regenerated. When the annotation verification information carries the second hint information, the global view, the local view and the second hint information are input into the image annotation model to obtain the updated local annotation result of the sample image.

[0090] In one implementation, the first prompt message and the second prompt message can be implemented based on the annotation assistance plug-in for generating auxiliary descriptions installed in the terminal device. For example, the plug-in can be installed in the terminal device, and the checker can start the plug-in during the process of checking the annotation result.

[0091] In an alternative way, an Optical Character Recognition (OCR) plug-in can also be installed in the terminal device. The checker can interactively mark the text area in the sample image and draw a rectangular box. Then, the coordinates of the rectangular box can be input to the OCR plug-in to output the corresponding text content. Based on the text content, corresponding annotation results can be added, and a sample set can be constructed by combining the annotation results with the aforementioned global annotation results and local annotation results. For example, the text content can be directly used as a newly added local annotation result, and the local annotation result can be merged with the originally obtained local annotation results to construct the sample set. In an alternative way, based on the text content, certain adjustments can be made to the originally obtained global annotation results and local annotation results, and a sample set can be constructed based on the adjusted annotation results.

[0092] Taking the model to be trained as a question-and-answer model as an example, the specific construction method of the sample set will be introduced below. In the embodiments of this specification, the construction device can construct question-and-answer pairs based on the obtained annotation results based on a preset prompt message template, and use the obtained question-and-answer pairs as the sample data for training the question-and-answer model. These prompt message templates can be standard prompt message templates historically used in the industry, or can be prompt message templates generated by the model based on industry requirements.

[0093] In some embodiments, the step of constructing a sample set based on the sample image, the global annotation result, and the local annotation result in step 208 includes:

[0094] Filling the global annotation result and the local annotation result into the target prompt message template; wherein, the target prompt message template includes a content part, an instruction part, and an output part. The content part indicates the information of the target task to be performed by the question-and-answer model and the scenario it targets. The instruction part indicates the task instruction to be performed by the question-and-answer model. The global annotation result and the local annotation result are filled into the output part;

[0095] Generating question-and-answer pairs based on the information in the filled target prompt message template, and constructing a sample set for training the question-and-answer model based on the question-and-answer pairs; wherein, the question information in the question-and-answer pair includes the information of the content part and the instruction part, and the answer information in the question-and-answer pair includes the information of the output part.

[0096] In the embodiments of this specification, each prompt information template to be selected may include a context part, an instruction part, and an output part. The context part indicates the information about the target task to be executed by the question-and-answer model and the scenario it targets. The instruction part indicates the task instructions for the question-and-answer model to execute. The global annotation result and the local annotation result are filled into the output part. Keywords corresponding to the industry or task are established in each template. For example, a prompt information template includes context: "This is a {scenario category} scenario, mainly focusing on the following events: {task requirement information}", instruction: "Are there any obvious abnormal events in this scenario?", output: "I found the following abnormal events: {annotation content}". The "{scenario category}" is used to fill in the information about the scenario targeted by the question-and-answer model. The "{task requirement information}" is used to fill in the requirements information related to the target task to be executed by the question-and-answer model. The "{annotation content}" is used to fill in the global annotation result and the local annotation result obtained by annotating the sample image. The scenario category and task requirement information can be given by the user.

[0097] After obtaining the annotation result for the sample image, the construction device can fill the annotation result into the selected target prompt template, and based on the information in the template, it can be automatically assembled and organized into the form of a question-and-answer pair. Such as the format {Q: Context + Instruction; A: Output}, where Q represents the question information and A represents the answer information.

[0098] In some embodiments, multiple local annotation results are generated by annotating the sample image. The construction device can summarize these multiple local annotation results, such as by using a large model for summarization, and generate a more rich and comprehensive overall description information. Then, based on this overall description information, question-and-answer pairs are generated to ensure the richness and comprehensiveness of the answer information.

[0099] In the embodiments of this specification, before constructing the question-and-answer pair using the prompt information template, it is necessary to first determine the target prompt information template to be used for the sample image. In some embodiments, before filling the global annotation result and the local annotation result into the target prompt information template, the sample set construction method provided in the embodiments of this specification further includes: querying for alternative prompt information templates in multiple prompt information templates whose similarity to the task requirement information is higher than the target threshold; where the task requirement information indicates the target task to be executed by the question-and-answer model; in the case of querying at least one alternative prompt information template, any one of the alternative prompt information templates is used as the target prompt information template.

[0100] The construction device can determine the target prompt information template based on the task requirement information. Among the multiple prompt information templates to be selected, each prompt information template can correspond to an applicable field, scenario, and task. For example, in the content part of each prompt information template, corresponding information is pre-filled at the "scenario category" and "task requirement information". The construction device can compare the task requirement information corresponding to the model to be trained with the information in each prompt information template to determine the similarity. When the similarity is higher than the set target threshold, the corresponding prompt information template is used as a backup prompt information template. In one way, the construction device compares the scenario to which the model to be trained will be applied and the corresponding task requirement information with the scenarios and task requirement information corresponding to each prompt information template to determine the similarity. For the model to be trained, when using each sample image for annotation and constructing question-answer pairs, the target prompt information template can be selected from the determined backup prompt information templates here, without having to determine the corresponding alternative prompt information template for each sample image.

[0101] If only one backup prompt template is determined, directly determine this backup prompt information template as the target prompt template corresponding to each sample image. In the case where multiple backup prompt templates are determined, for each sample image, any one of the multiple backup prompt templates can be randomly selected as the target prompt information template. This can ensure that the forms or expressions of the question-answer pairs constructed for different sample images are different, guarantee the flexibility of constructing the question-answer pairs, avoid rigid presentation forms, obtain sample data with more expression ways, and ensure better training effects for the model.

[0102] In some embodiments, before filling the global annotation result and the local annotation result into the target prompt information template, the sample set construction method provided in the embodiments of this specification further includes: filling the task requirement information and the information of the scenario it targets into the content part of the target prompt information template, and using a text adjustment model to adjust the content in the target prompt information template.

[0103] The construction device can obtain the task requirement information corresponding to the model to be trained and the information about the scenario it targets, and fill these information into the "task requirement information" and "scenario category" in the target prompt information template respectively. For example, the original information in the "task requirement information" and "scenario category" in the target prompt information template can be replaced with the task requirement information corresponding to the model to be trained and the information about the scenario it targets. In some embodiments, the construction device can also perform enhancement processing on the target prompt template to ensure that the filled template information is more sufficient and rich. For example, the construction device can obtain industry background knowledge for the model to be trained and fill this industry background knowledge into the content part of the template. For example, add a "background knowledge" part after the "scenario category" and fill the obtained industry background knowledge into this position.

[0104] The construction device can also use a text adjustment model to perform enhancement processing on the target prompt template to improve the fluency and integrity of the template information, and can perform a small amount of rewriting on the template information to make the rewritten information more consistent with the filled scenario category and task requirement information. This text adjustment model can be a large language model.

[0105] In some embodiments, the sample set used for constructing the training Q&A model based on Q&A pairs includes at least one of the following steps:

[0106] When the Q&A pair meets the complex Q&A conditions, disassemble the Q&A pair to obtain multiple sub-Q&A pairs; construct the sample set used for training the Q&A model based on the multiple sub-Q&As; wherein, the complex Q&A conditions include a complexity greater than the target threshold, and / or, the format meets the set Q&A format;

[0107] Use a text adjustment model to rewrite the Q&A pair and keep the semantics of the rewritten Q&A pair unchanged; construct the sample set used for training the Q&A model based on the rewritten Q&A pair;

[0108] When the reply information in the Q&A pair needs to be inferred from the question information, generate the thinking chain information on which the reply information is based for the question information; construct the sample set used for training the Q&A model based on the thinking chain information and the reply information.

[0109] In one way, the construction device can disassemble the generated Q&A pairs. For some Q&A pairs that meet the complex Q&A conditions (such as long descriptions and complex content), a large model can be used to disassemble them into multiple sub-Q&A pairs. For example, according to the description of the original Q&A pair, the large model can automatically generate appropriate questions and answers. Construct the sample set used for training the Q&A model based on these multiple sub-Q&As. For example, use one sub-Q&A pair as a sample data in the sample set.

[0110] The complex Q&A conditions can be preset. For example, set a target threshold corresponding to the question complexity, and the complex Q&A conditions include a complexity greater than the target threshold. The question complexity can be determined by aspects such as the number of characters in the question, the number of fields involved, the amount of background information required, and the output requirements. Another example is to set at least one Q&A format belonging to complex questions, and the format of the complex Q&A conditions conforms to any one of the at least one Q&A format. A possible Q&A format is "The question information includes X question, does the xx content in the X question mean xxx, if so then how to xxx for xxx, if not then xxxxx; The answer information includes: Since xxxx, so xx in this question is xxx, further xxx, in addition xxx". The positions of the "X" and "x" symbols in the example Q&A format can be replaced with actual information, and this example is only used to represent the description format of the information. There may be other possible Q&A formats, or the actual Q&A format can also be different from this example, which is not limited here.

[0111] The Q&A pairs that meet the complex Q&A conditions can be specified by the staff, such as being pre-specified when constructing the question; or it can also be automatically judged by the construction device based on the set complex Q&A conditions, such as the complex Q&A conditions including that the number of characters contained in the question information exceeds the character number threshold, and the content to be queried in the question information exceeds the target quantity, etc.

[0112] In one way, the construction device can perform diversity evolution processing on the generated Q&A pairs to avoid the situation where the Q&A pairs are obtained by template filling and may have rigid questions and answers. For example, the large model can rewrite the Q&A pairs without changing the original sentence meaning of the Q&A pairs, rewrite them into more complex, diverse and smooth Q&A pairs, and then construct a sample set for training the Q&A model based on the rewritten Q&A pairs.

[0113] In one way, the construction device can perform thought chain extension processing on the generated Q&A pairs. For example, for the question information in the Q&A pair where the answer cannot be directly obtained and reasoning is required for the question information to obtain the answer information, guide the large model to generate a thinking process or a multi-step reasoning process for the question information, determine the thought chain information based on which the answer information is obtained; and then use the obtained thought chain information as sample data together to construct a sample set for training the Q&A model.

[0114] After obtaining the Q&A pairs required for training the Q&A model based on the annotation results, the construction device can also perform data evolution processing on these Q&A pairs in these ways, so that the quality of the processed Q&A pairs is better and more conducive to model training.

[0115] In some embodiments, after obtaining the annotation result for the sample image, the construction device can also screen the annotation result to ensure that better-quality annotation results are put into use for model training, avoiding generating poor outputs by the model. In step 208 above, based on the sample image, the global annotation result, and the local annotation result, a sample set is constructed, including: when the global annotation result and the local annotation result meet the screening conditions, constructing a sample set based on the sample image, the global annotation result, and the local annotation result; wherein, the screening conditions include that the content is compliant and / or the corresponding perplexity (PPL, Perplexity) is within the target range.

[0116] The construction device can, based on the screening conditions, remove harmful content, illegal content, and content suspected of bad guidance in the global annotation result and the local annotation result, etc. Exemplarily, the construction device can use a large model to perform this screening process.

[0117] Perplexity is an important indicator for measuring the performance of a language model, mainly used to evaluate the prediction ability of the model for text data, reflecting the degree of uncertainty of the model when generating or predicting text. The lower the perplexity, the more accurate the model's prediction of the text; conversely, the higher the perplexity, the weaker the model's prediction ability. For images with low perplexity and their corresponding annotation results, it can be determined that the model itself can handle such data well, so there is no need to further train the model based on these data to avoid wasting resources. For data with too high perplexity, it may be that there is a problem with the data itself and it is not suitable for model training. The sample set may not be constructed based on this data, and this data may not be used for model training. In some embodiments, for data with too high perplexity, it may also be that the model does not have the ability to process such data. It can be determined whether to use this data as sample data by manually judging whether the model needs to train the corresponding ability.

[0118] In some embodiments, the construction device can also screen the input sample image, and filter out images with abnormal resolution, aspect ratio, blurred images, overexposed or underexposed images, and not perform subsequent annotation processes on these images.

[0119] Figure 3 It is the overall implementation framework diagram of a sample set construction method provided by an embodiment of this specification. Figure 3It can be cross-referred with the foregoing introduction to the sample set construction method. This solution mainly includes stages such as data intelligent annotation, instruction template construction, and data evolution. In the data intelligent annotation stage, the input is the original image, and the output is the overall and detailed annotation text of the image. It can also output detection boxes or polygons corresponding to the target object, etc. The input of the instruction template construction stage is the prompt template library, the actual task scenario and task requirements, industry background knowledge, etc., and the output is multiple alternative prompt templates. The input of the data evolution stage is the basic Q&A pairs spliced by the template and the annotation results, and the output is the evolved sample set.

[0120] As Figure 3 shown, information such as the original image (such as the foregoing sample image), the prompt template library, the scenario / task requirement information corresponding to the model, and industry background knowledge can be provided to the construction device first. The construction device can perform overall pre-annotation on the original image to generate a global annotation result. It can also detect and segment the target object in the original image, process the segmented detailed image (i.e., the foregoing local view), and perform detailed pre-annotation to obtain a local annotation result. Then, the obtained annotation result can be verified by manual annotation. During the manual annotation verification process, the verification personnel can use intelligent auxiliary annotation plugins such as an interactive target segmentation plugin, an auxiliary description generation plugin, and an OCR plugin to perform quick annotation updates.

[0121] The construction device can retrieve alternative prompt templates that meet the requirements from the prompt template library based on the scenario / task requirement information corresponding to the model, and enhance the alternative prompt templates based on industry background knowledge to improve the fluency and richness of the template information. The construction device fills the annotation result obtained after verification into the target prompt template selected from the alternative prompt templates to construct basic Q&A pairs. Data evolution can be performed on the basic Q&A pairs to obtain higher-quality Q&A pairs, and then a sample set can be constructed for training the model. This data evolution can include methods such as Q&A pair disassembling, diversity evolution, thought chain extension, and filtering of low-quality data. For these methods, reference can be made to the foregoing relevant introduction, and details will not be elaborated here.

[0122] Figure 3 The image detection / segmentation, detailed image processing, interactive target segmentation, and OCR processes in can be implemented using a computer vision (CV, Computer Vision) module. The overall pre-annotation, detailed pre-annotation, and auxiliary description generation processes can be implemented using a vision language model. The processes of prompt template enhancement, Q&A pair disassembling, diversity evolution, thought chain extension, and filtering of low-quality data can be implemented using a large model.

[0123] In the embodiments of this specification, based on image detection and segmentation, multi-modal large models, and visual prompting techniques, intelligent annotation of the overall and detailed content of sample images can ensure the high-quality generation of annotation results. This annotation process also supports a variety of interactive intelligent auxiliary annotation plugins to facilitate the verification of annotation results using these plugins, which can improve the efficiency of manual verification and modification of annotations, reduce the annotation cost significantly while ensuring the quality of annotation results, especially enhancing the comprehensiveness and detailed description quality of complex image descriptions.

[0124] In addition, in the embodiments of this specification, by filling in information such as annotation results and task requirements in a semi-structured prompt information template, question-answer pairs required for training the model are constructed. It is possible to retrieve appropriate industry instruction templates based on industry scenarios, task objectives, etc., and drive the large model to perform template enhancement and fill in annotation results, thereby constructing a question-answer dataset more suitable for improving the fine-tuning ability of the large model under target scenario tasks. It is also possible to use various data evolution operators such as question-answer pair decomposition, diversity evolution, and thought chain extension to improve the quality and diversity of data and ensure better construction of the sample set.

[0125] In summary, in the sample set construction method provided in the embodiments of this specification, on the basis of annotating the global view of the sample image, object recognition of the sample image and the interception of the corresponding local view are also performed. Visual prompt information is used for the global view and the local view to instruct the image annotation model to perform local annotation on the image, and a sample set is constructed based on the global annotation result and the local annotation result. In this way, more comprehensive and complete annotation of the sample image can be ensured, and the accuracy and comprehensiveness of the annotation result can be improved. Correspondingly, the sample set obtained based on this annotation result can achieve better training effects on the model and ensure a better-performing model.

[0126] The sample set constructed in the embodiments of this specification can be used to train a question-answer model. Figure 4 It is a flowchart of a training method for a question-answer model provided in an embodiment of this specification. This training method can be applied to a model training device, which can be the same as or different from the sample set construction device that executes the above sample set construction method. For example, this training method can be executed by a cloud platform. When the model training device is different from the sample set construction device, after the construction device completes the training of the question-answer model by the model training device, the sample set construction device sends the constructed sample set to the model training device, and the model training device trains the question-answer model based on this sample set. As Figure 4 shown, the training method of the question-answer model includes the following steps 402 to 406.

[0127] Step 402, obtain a sample set, where the sample set is constructed using the above sample set construction method.

[0128] Step 402 may refer to the relevant introduction for Figure 2 and Figure 3 . The sample set may include multiple sample data, and each sample data may include a sample image and a corresponding annotation result, where the annotation result includes a global annotation result and a local annotation result.

[0129] Step 404: Input the sample images in the sample set and the sample question information for the sample images into the initial question-and-answer model to predict the reference answer information for the sample question information.

[0130] The initial question-and-answer model can analyze the sample images based on the input sample question information and predict the reference answer information for the sample question information. In some embodiments, the initial question-and-answer model is a question-and-answer model that has undergone a certain pre-training process. Fine-tune the question-and-answer model for a specific industry based on the data in the aforementioned sample set to ensure that the question-and-answer model can be applied to accurate text-image questions and answers in that industry.

[0131] Step 406: Train the question-and-answer model based on the reference answer information, the global annotation result, and the local annotation result of the sample image.

[0132] For the reference answer information of each sample image output by the initial question-and-answer model, it can be compared with the global annotation result and the local annotation result of the sample image, and the parameters of the initial question-and-answer model can be adjusted based on the comparison result. Then, repeatedly execute the processes of prediction, comparison, and parameter adjustment until the training stop condition is reached to obtain the trained question-and-answer model. Specifically, the loss value can be calculated according to the prediction result (i.e., the reference answer information of the sample question information) and the annotation result of the sample image, and the model parameters of the initial question-and-answer model can be adjusted by backpropagation based on the loss value. Then, return to select the sample image and the sample question information and train the initial question-and-answer model until the training stop condition is reached. The training is completed to obtain a question-and-answer model that can be put into use. Among them, the training stop condition can be that the loss value is less than the loss value threshold, or it can also be that the number of iterations reaches the number threshold, or it can also be combined with the loss value threshold and the number threshold to determine whether the training stop condition is reached.

[0133] The loss value threshold is the critical value of the loss value, and the specific value of the loss value threshold is selected according to the actual situation. When the loss value is greater than or equal to the loss value threshold, it indicates that there is still a certain deviation between the prediction result of the initial question-answering model and the true result, and the model parameters of the initial question-answering model still need to be adjusted. At this time, it is determined that the training stop condition has not been reached. When the loss value is less than the loss value threshold, it indicates that the proximity between the prediction result of the initial question-answering model and the true result is sufficient, and the training can be stopped. At this time, it can be determined that the training stop condition has been reached. In addition, the number threshold is set according to the actual situation. When the number of iterations reaches the number threshold, it means that the number of training times of the initial question-answering model is sufficient. At this time, the prediction result of the initial question-answering model is already close to the true result, and the training can be stopped.

[0134] In the embodiments of this specification, the image annotation model used during the execution of the sample set construction method can directly adopt existing image annotation models on the market, or can also be obtained by self-training on a cloud platform.

[0135] In summary, based on annotating the global view of the sample image, object recognition and corresponding local view extraction are also performed on the sample image. For the global view and local view, visual cue information is used to instruct the image annotation model to perform local annotation on the image, and a sample set is constructed based on the global annotation result and the local annotation result. In this way, more comprehensive and complete annotation of the sample image can be ensured, and the accuracy and comprehensiveness of the annotation result can be improved. Correspondingly, the sample set obtained based on this annotation result can achieve a better training effect on the model, and ensure that a question-answering model with better performance is obtained.

[0136] Utilize Figure 4 The question-answering model trained can be applied to the question-answering process. Correspondingly, based on this question-answering model, a better question-answering processing effect can be obtained, and a more accurate and comprehensive answer effect can be obtained for the question information. Figure 5 is a flowchart of a question-answering processing method provided by an embodiment of this specification. As Figure 5 shown, this method includes the following steps 502 and 504. Step 502: Obtain target question information for a target image. Step 504: Input the target image and the target question information into the question-answering model to obtain the answer information of the target question information output by the question-answering model; wherein, the question-answering model is trained based on the above training method.

[0137] The question and answer processing method provided in the embodiments of this specification can be applied to a cloud platform. As described above, if the question and answer processing device is a cloud platform, the above target question information can be sent by a terminal device. After obtaining the answer information to the target question information output by the question and answer model, the cloud platform can send the answer information to the terminal device for the terminal device to display. In some embodiments, the terminal device can also send question feedback information regarding the obtained answer information to the cloud platform. The cloud platform can adjust the question and answer model based on the question feedback information.

[0138] Figure 6 It is a flowchart of a request processing method based on a question and answer model provided in an embodiment of this specification, which is applied to a cloud platform and specifically includes the following steps 602 and step 604.

[0139] Step 602: Receive a model request sent by a terminal device, where the model request includes at least one piece of information among the scene identifier of the target scene, the scene input data of the target scene, and the target model specification parameters.

[0140] The cloud platform can be an online platform that provides model services, allowing developers, enterprises, or individual users to submit tasks, obtain resources, or implement specific functions through API calls, web interfaces, or other interaction methods. In the embodiments of this specification, the cloud platform is a cloud service platform capable of providing deep learning models, allowing external applications (such as e-commerce platforms, navigation applications, social applications, film and television production applications, and online education platforms, etc.) to access and use. The cloud platform manages the training, update, optimization, load balancing, scheduling, and inference of models to ensure efficient and accurate responses to requests from terminal devices. The terminal device is the terminal device that requests to obtain the question and answer model.

[0141] In actual implementation, the model request is sent by the terminal device to the cloud platform and is a request aimed at obtaining or using a specific model service. The model request can include at least one piece of information among the scene identifier of the target scene, the scene input data of the target scene, and the target model specification parameters. In the embodiments of this specification, the model request can be a request for the question and answer model.

[0142] Among them, the target scenario is the specific environment or situation where the question-and-answer model will be applied, which describes the purpose and expected effect of the model service. The scenario identifier of the target scenario is the identification information used to identify the target scenario, which is used to assist in quickly locating the model related to a specific scenario. For example, "data annotation", "document question-and-answer", "picture question-and-answer", "video question-and-answer", "video recommendation", "music recommendation", "news recommendation", "video navigation", "video surveillance", "video quality evaluation", "object positioning", "product positioning and search", and so on. The scenario input data of the target scenario can be the input data used to train the pre-trained question-and-answer model in the target scenario to make it adapt to the target scenario. For example, in the picture question-and-answer scenario, the scenario input data can be sample images with reply information. In addition, the model specification parameters are the attribute specification parameters that define the model architecture, performance, or configuration, such as model type, model size, etc. The model specification parameters are used to determine the model that adapts to the user's needs.

[0143] For example, the user clicks on the scenario identifier of the target scenario as "picture question-and-answer" on the terminal device, selects the required model size as 32GB, and uploads the scenario input data of the target scenario. The terminal device generates a model request and sends it to the cloud platform. The cloud platform receives the model request sent by the terminal device to the cloud platform. Among them, the model request includes the scenario identifier of the target scenario, the scenario input data of the target scenario, and the target model specification parameters.

[0144] In the embodiments of this specification, the user can independently select at least one piece of information from the scenario identifier of the target scenario, the scenario input data of the target scenario, and the target model specification parameters on the terminal device to trigger a model request. The cloud platform can receive the model request sent by the terminal device for data annotation and provide reference information for selecting a question-and-answer model based on at least one of the scenario identifier of the target scenario, the scenario input data of the target scenario, and the target model specification parameters, with a wider applicable scenario.

[0145] Step 604: Determine the corresponding question-and-answer model from at least one model based on the model request, where the at least one model is trained by the above-mentioned training method of the question-and-answer model, and the question-and-answer model is used to obtain the reply information of the target question information.

[0146] In the embodiments of this specification, a model library may be provided. The model library may store at least one question-and-answer model adapted to different scenarios, as well as question-and-answer models with multiple different model specification parameters. For a model request, to determine a corresponding question-and-answer model from at least one model, an optional method is: based on the model request, search for the corresponding question-and-answer model from at least one model included in the model library, and the model library may store question-and-answer models with multiple different model specification parameters; another optional method is: based on the model request, train to obtain a question-and-answer model; still another optional method is: based on the model request, construct a question-and-answer model, which is not limited here.

[0147] In one implementation of step 604, the above-mentioned determining a corresponding question-and-answer model from at least one model based on the model request may include:

[0148] Based on at least one piece of information in the model request, search for the question-and-answer model corresponding to the at least one piece of information in at least one model in the model library; among them, the scenario identifier of the target scenario and the scenario input data of the target scenario correspond to the model adapted to the target scenario, and the target model specification parameters correspond to the model with the same model specification parameters as the target model specification parameters;

[0149] When at least one piece of information includes the scenario input data of the target scenario, based on the scenario input data of the target scenario, train the found question-and-answer model to obtain a trained question-and-answer model.

[0150] For the above implementation of step 604, the cloud platform may, for the scenario identifier of the target scenario and the target model specification parameters, adopt the method of searching for a model in the model library to determine the required question-and-answer model. For the scenario input data of the target scenario, adopt the method of searching for a model in the model library and continuing to train the found model to determine the required question-and-answer model.

[0151] For the case where the model request includes only one piece of information, if the model request includes the scenario identifier of the target scenario, step 604 may include: based on the scenario identifier of the target scenario, search for the question-and-answer model adapted to the target scenario from the model library, where the model library stores at least one question-and-answer model adapted to different scenarios. If the model request includes the scenario input data of the target scenario, step 604 may include: determine an initial question-and-answer model adapted to the target scenario from at least one model; based on the scenario input data of the target scenario, train the initial question-and-answer model to obtain a trained question-and-answer model. If the model request includes model specification parameters, step 604 may include: based on the model specification parameters, search for the corresponding question-and-answer model from the model library, where the model library stores question-and-answer models with multiple different model specification parameters.

[0152] When two or three items of information including the scene identifier of the target scene, the scene input data of the target scene, and the target model specification parameters are included in the model request, the cloud platform can search for a model in the model library that simultaneously meets the two or three items of information to determine the required model. If the model request includes the scene identifier of the target scene and the target model specification parameters, the cloud platform can search for a model corresponding to the target model specification parameters adapted to the target scene in the model library. If the model request includes the scene identifier of the target scene and the scene input data of the target scene, the cloud platform can first search for a model adapted to the target scene in the model library, and then train the model based on the scene input data of the target scene to obtain the required model. If the model request includes the target model specification parameters and the scene input data of the target scene, the cloud platform can first search for a model with the target model specification parameters in the model library, and then train the model based on the scene input data of the target scene to obtain the required model.

[0153] For the case where the model request includes the scene identifier of the target scene, the scene input data of the target scene, and the target model specification parameters at the same time, the cloud platform can first search for at least one pre-trained question-and-answer model in the model library based on the scene identifier of the target scene, then screen out the question-and-answer model with the target model specification parameters from the at least one question-and-answer model, and then train the question-and-answer model with the target model specification parameters based on the scene input data of the target scene to obtain a question-and-answer model suitable for the user's needs.

[0154] The question-and-answer model obtained by the cloud platform is trained according to Figure 4 the training method of the above-mentioned question-and-answer model provided, and the specific method can be referred to the content of the above model training, which will not be elaborated here.

[0155] In the embodiments of this specification, based on the scene requirements, the question-and-answer models adapted to the scene can be accurately found through the scene identifier, and then the reply information corresponding to the question information can be generated by using these models, which can ensure that the reply information corresponding to the obtained question information is more accurate and scene-appropriate, thus improving the user experience and the quality of the question and answer.

[0156] Optionally, for question and answer in different scenarios, there may be certain differences in the reply information corresponding to the required question information. As an example, the cloud platform can provide question-and-answer models in multiple scenarios, and can provide corresponding question-and-answer models based on the model request for data annotation sent by the terminal device, so as to use the question-and-answer model to determine the accurate and effective reply information corresponding to the target question information.

[0157] In the embodiments of this specification, after step 604, the cloud platform may further deploy a question-and-answer model, and based on the question-and-answer model, construct a call interface (for example, the call interface may be a question-and-answer interface), so that the terminal device can call the question-and-answer model through the call interface to obtain a reply to the target question information. Exemplarily, the cloud platform may deploy the question-and-answer model in a virtual machine, a container orchestration platform, an edge node, or a dedicated processor. When the terminal device makes a call, it triggers the device on which the model is deployed to run the corresponding program or code to implement the reply to the target question information.

[0158] Corresponding to the above method embodiments, this specification also provides embodiments of a task platform. Figure 7 It is a schematic structural diagram of a task platform provided by an embodiment of this specification. As Figure 7 shown, the task platform 700 includes: a request interface 702 and a response unit 704;

[0159] The request interface 702 is used to receive a model request sent by a terminal device, where the model request includes at least one of a scene identifier of a target scene, scene input data of the target scene, and model specification parameters;

[0160] The response unit 704 is used to determine a corresponding question-and-answer model from at least one model based on the model request, where the at least one model is trained by the above-mentioned training method of the question-and-answer model.

[0161] Optionally, the model library stores at least one question-and-answer model adapted to different scenarios and question-and-answer models with multiple different model specification parameters; the response unit 704 is used to: based on at least one piece of information in the model request, search for a question-and-answer model corresponding to at least one piece of information in the model library; where the scene identifier of the target scene and the scene input data of the target scene correspond to the model adapted to the target scene, and the target model specification parameters correspond to the model with the same model specification parameters as the target model specification parameters; in the case where at least one piece of information includes the scene input data of the target scene, train the found question-and-answer model based on the scene input data of the target scene to obtain a trained question-and-answer model.

[0162] Optionally, the task platform further includes a call interface, which is constructed based on the question-and-answer model; the call interface is used for the terminal device to call to obtain a reply to the target question information.

[0163] The above is a schematic solution of a task platform in this embodiment. It should be noted that the technical solution of this task platform and the technical solution of the above sample set construction method belong to the same concept. For the details not described in the technical solution of the task platform, reference can be made to the description of the technical solution of the above question-and-answer processing method.

[0164] In some embodiments, there may not be an inference model (i.e., a question-and-answer model) capable of executing the question-and-answer processing method. In this case, it is necessary to perform basic training on some existing models to obtain the inference model. Embodiments of this specification also provide a model training platform, where the required inference model can be obtained through model training on the model training platform.

[0165] The model training platform includes a request interface unit, a model training unit, and a response unit. This model training platform can refer to the relevant introduction to the task platform above. The request interface unit is used to receive a model training request, where the model training request includes model attribute information of the machine learning model to be trained; the model training unit is used to determine the machine learning model to be trained according to the model attribute information, and perform model training on the machine learning model to be trained to obtain a target model, where the target model is obtained by training based on the above training method. The response unit is used to output the target model.

[0166] Optionally, the model training platform further includes a data receiving unit, which is used to receive training data input by the user and send the training data to the model training unit. The model training unit is also used to perform model training on the target model according to the training data to obtain a trained target model. Optionally, the model training platform further includes a model library, where multiple machine learning models are stored; the model training unit is also used to determine the machine learning model to be trained from the model library according to the model attribute information.

[0167] Embodiments of this specification also provide a model training method, which is applied to a model training platform and includes: receiving a model training request, where the model training request includes model attribute information of the machine learning model to be trained; determining the machine learning model to be trained according to the model attribute information, and performing model training on the machine learning model to be trained to obtain a target model, where the target model is obtained by training through the above model training method; outputting the target model.

[0168] Corresponding to the above embodiments of the sample set construction method, this specification also provides embodiments of a sample set construction device. Figure 8 It is a schematic structural diagram of a sample set construction device provided by an embodiment of this specification. As Figure 8 shown, the sample set construction device includes:

[0169] The first acquisition module 802 is used to input a sample image into an image annotation model to obtain a global annotation result of the sample image;

[0170] The second acquisition module 804 is used to identify the target object in the sample image, and based on the obtained recognition result, mark the target object in the sample image to obtain a global view and intercept the target object to obtain a local view;

[0171] A third acquisition module 806, configured to input a global view, a local view, and visual hint information into an image annotation model to obtain a local annotation result of a sample image, where the visual hint information is used to prompt the image annotation model to perform an annotation task on the global view based on the local view;

[0172] A first construction module 808, configured to construct a sample set based on the sample image, the global annotation result, and the local annotation result.

[0173] Optionally, the sample set construction device further includes:

[0174] A second construction module, configured to construct preset hint information based on task requirement information before inputting the sample image into the image annotation model to obtain a global annotation result of the sample image; where the task requirement information indicates a target task to be performed by the model to be trained; the preset hint information is used to prompt the image annotation model to extract information of the input image according to a target content type corresponding to the target task for global annotation;

[0175] A first acquisition module 802 is configured to: input the sample image and the preset hint information into the image annotation model, so that the image annotation model extracts information of the sample image according to a target content type corresponding to the target task, and obtain a global annotation result of the sample image, where the global annotation result includes information of the target content type.

[0176] Optionally, the sample set construction device further includes: a third construction module, configured to construct visual hint information based on task requirement information before inputting the global view, the local view, and the visual hint information into the image annotation model, where the task requirement information indicates a target task to be performed by the model to be trained, the visual hint information is used to prompt the image annotation model to perform an annotation task, extract information of the input local view according to a target content type corresponding to the target task, and determine association information between the input local view of the image and the global view.

[0177] Optionally, the first construction module 808 is configured to: send the sample image, the global annotation result, and the local annotation result to the front end, and receive annotation verification information for the sample image sent by the front end; in the case where the annotation verification information indicates that the local annotation result fails the verification and carries region information of the region where the target object is located, intercept an auxiliary region in the sample image indicated by the region information to obtain an updated local view; input the global view, the updated local view, and the visual hint information into the image annotation model to obtain an updated local annotation result of the sample image; where the visual hint information is used to prompt the image annotation model to perform an annotation task on the global view based on the updated local view; construct a sample set based on the sample image, the global annotation result, and the new local annotation result.

[0178] Optionally, the first construction module 808 is configured to:

[0179] Send the sample image, the global annotation result, and the local annotation result to the front end, and receive the annotation verification information for the sample image sent by the front end;

[0180] In the case where the annotation verification information indicates that the global annotation result fails the verification and carries the first prompt information, input the sample image and the first prompt information into the image annotation model to obtain the updated global annotation result of the sample image; wherein, the first prompt information is used to prompt the image annotation model to extract the information of the input image according to the target content type for global annotation;

[0181] In the case where the annotation verification information indicates that the local annotation result fails the verification and carries the second prompt information, input the global view, the local view, and the second prompt information into the image annotation model to obtain the updated local annotation result of the sample image; wherein, the second prompt information is used to prompt the image annotation model to perform the annotation task on the global view based on the local view, and the second prompt information is different from the visual prompt information;

[0182] Construct a sample set based on the sample image, the updated global annotation result, and the updated local annotation result.

[0183] Optionally, the first construction module 808 is configured to:

[0184] Fill the global annotation result and the local annotation result into the target prompt information template; wherein, the target prompt information template includes a content part, an instruction part, and an output part, the content part indicates the information of the target task to be performed by the question-and-answer model and the scenario it targets, the instruction part indicates the task instruction to be performed by the question-and-answer model, and the global annotation result and the local annotation result are filled into the output part;

[0185] Generate question-and-answer pairs based on the information in the filled target prompt information template, and construct a sample set for training the question-and-answer model based on the question-and-answer pairs; wherein, the question information in the question-and-answer pairs includes the information of the content part and the instruction part, and the answer information in the question-and-answer pairs includes the information of the output part.

[0186] Optionally, the sample set construction device further includes:

[0187] A query module, configured to query, among multiple prompt information templates, a backup prompt information template whose similarity to the task requirement information is higher than the target threshold before filling the global annotation result and the local annotation result into the target prompt information template; wherein, the task requirement information indicates the target task to be performed by the question-and-answer model;

[0188] A determination module, configured to use any one of the backup prompt information templates as the target prompt information template when at least one backup prompt information template is queried.

[0189] Optionally, the sample set construction device further includes: a rewriting module, configured to fill the task requirement information and the information of the scenario it targets into the content part of the target prompt information template before filling the global annotation result and the local annotation result into the target prompt information template, and use a text adjustment model to adjust the content in the target prompt information template.

[0190] Optionally, the first construction module 808 is configured to perform at least one of the following steps:

[0191] When the question-and-answer pair meets the complex question-and-answer condition, disassemble the question-and-answer pair to obtain multiple sub-question-and-answer pairs; construct a sample set for training the question-and-answer model based on the multiple sub-questions; wherein, the complex question-and-answer condition includes that the complexity is greater than the target threshold, and / or the format meets the set question-and-answer format;

[0192] Use a text adjustment model to rewrite the question-and-answer pair and keep the semantics of the rewritten question-and-answer pair unchanged; construct a sample set for training the question-and-answer model based on the rewritten question-and-answer pair;

[0193] When the reply information in the question-and-answer pair needs to be inferred based on the question information, generate the thinking chain information on which the reply information is obtained based on the question information; construct a sample set for training the question-and-answer model based on the thinking chain information and the reply information.

[0194] Optionally, the first construction module 808 is configured to: when the global annotation result and the local annotation result meet the screening condition, construct a sample set based on the sample image, the global annotation result and the local annotation result; wherein, the screening condition includes that the content is compliant and / or the corresponding perplexity is within the target range.

[0195] The sample set construction device according to the embodiments of this specification can ensure more comprehensive and complete annotation of the sample image, improve the accuracy and comprehensiveness of the annotation result. Correspondingly, the sample set obtained based on this annotation result can achieve a better training effect on the model and ensure a better-performing model.

[0196] Figure 9 It is a schematic structural diagram of a training device for a question-and-answer model provided by an embodiment of this specification. This training device can be Figure 4 mutually referred to with the Figure 9 shown training method. As

[0197] The sample set acquisition module 902 is configured to acquire a sample set, wherein the sample set is constructed by using the above-mentioned sample set method;

[0198] A prediction module 904, configured to input a sample image in a sample set and sample question information for the sample image into an initial question-answering model, and predict reference answer information for the sample question information;

[0199] A training module 906, configured to train the question-answering model based on the reference answer information, the global annotation result and the local annotation result of the sample image.

[0200] Figure 10 It is a schematic structural diagram of a question-answering processing device provided by an embodiment of this specification. This question-answering processing device can be referred to Figure 5 mutually with the question-answering processing method shown. As Figure 10 shown, this question-answering processing device includes:

[0201] A question acquisition module 1002, configured to acquire target question information for a target image;

[0202] An answer module 1004, configured to input the target image and the target question information into the question-answering model, and obtain answer information for the target question information output by the question-answering model; wherein, the question-answering model is trained based on Figure 4 the training method shown.

[0203] Figure 11 It is a structural block diagram of a computing device provided by an embodiment of this specification. This computing device 1100 includes: a memory 1110 and a processor 1120; the memory 1110 is used to store computer programs / instructions, and the processor 1120 is used to execute the computer programs / instructions. When the computer programs / instructions are executed by the processor 1120, the steps of the above method are implemented. In one or more embodiments of this specification, this computing device can be understood as an integrated intelligent terminal, including but not limited to a server, a desktop computer, a PC (Personal Computer), a model all-in-one machine, a mobile phone, a tablet computer or other portable intelligent terminals, etc. And, this computing device may be pre-installed with the model described in the above embodiments of this application.

[0204] Specifically, the computing device can pre-set various types of models, including but not limited to models in the fields of natural language processing, visual processing, speech processing, code processing, multi-modal task processing, etc., so as to provide diverse model selections. In different product forms, the computing device can support one or more model usage methods, including but not limited to model training, model invocation, model fine-tuning, model deployment, model inference and application, etc. In some product forms, the computing device also supports model management, including but not limited to multi-type model management (supporting the management of various types of models such as discriminative and generative models), model version control (supporting the control of different model versions), model evaluation (evaluating the performance and effect of the model based on model evaluation tools), etc. In other product forms, the computing device can also create applications based on the model, provide API (Application Programming Interface) invocation capabilities, and can call the model into the created application through the API interface, while providing application management tools to realize the management and monitoring of the application.

[0205] Furthermore, the computing device can also include data management (supporting the creation and management of model tuning data sets), a training center (providing rich training resources to help users learn and master artificial intelligence AI technology), and basic control capabilities (providing enterprise-level basic control capabilities to ensure the security and efficient operation of the system). Through the above functions, a comprehensive and integrated AI development, training, deployment, and application device is provided.

[0206] Figure 12 It is a structural block diagram of an electronic device provided by an embodiment of this specification. The electronic device 1200 includes a memory 1210 and a processor 1220, and the memory 1210 and the processor 1220 are connected through a bus; the memory 1210 is used to store computer programs / instructions, and the processor 1220 is used to execute the computer programs / instructions. When the computer programs / instructions are executed by the processor 1220, the steps of the above method are implemented. Specifically, the components of the electronic device 1200 include but are not limited to the memory 1210 and the processor 1220. The processor 1220 and the memory 1210 can be connected through a bus 1230.

[0207] The electronic device 1200 may further include an access device 1240, which enables the electronic device 1200 to communicate with a database 1250 storing data via one or more networks 1260. Examples of such networks include the Public Switched Telephone Network (PSTN), Local Area Network (LAN), Wide Area Network (WAN), Personal Area Network (PAN), or a combination of communication networks such as the Internet. The access device may include one or more of any type of wired or wireless network interfaces (e.g., a network interface controller (NIC)), such as an IEEE802.11 Wireless Local Area Network (WLAN) wireless interface, Worldwide Interoperability for Microwave Access (Wi-MAX) interface, Ethernet interface, Universal Serial Bus (USB) interface, cellular network interface, Bluetooth interface, Near Field Communication (NFC).

[0208] In one embodiment of the present specification, the above components of the electronic device 1200 and Figure 12 other components not shown may also be connected to each other, for example, via a bus. It should be understood that Figure 12 the block diagram of the electronic device shown is for illustrative purposes only and is not a limitation on the scope of the present specification. Those skilled in the art may add or replace other components as needed. The electronic device 1200 may be any type of stationary or mobile electronic device, including a mobile computer or mobile electronic device (e.g., a tablet computer, personal digital assistant, laptop computer, notebook computer, netbook, etc.), a mobile phone (e.g., a smartphone), a wearable electronic device (e.g., a smartwatch, smart glasses, etc.) or other types of mobile devices, or a stationary electronic device such as a desktop computer or personal computer (PC). The electronic device 1200 may also be a mobile or stationary server.

[0209] One embodiment of the present specification also provides a computer-readable storage medium storing computer programs / instructions, which when executed by a processor implement the steps of the above method.

[0210] An embodiment of this specification also provides a computer program product, including computer programs / instructions, which, when executed by a processor, implement the steps of the above method.

[0211] The computer programs / instructions include computer program code, which can be in the form of source code, object code, executable files, or some intermediate forms, etc. The computer-readable medium can include: any entity or device capable of carrying the computer program code, recording medium, USB flash drive, mobile hard disk, magnetic disk, optical disc, computer memory, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), electrical carrier signals, telecommunication signals, and software distribution media, etc. It should be noted that the content included in the computer-readable medium can be appropriately increased or decreased according to the requirements of patent practice. For example, in some regions, according to patent practice, the computer-readable medium does not include electrical carrier signals and telecommunication signals.

[0212] The technical solutions of the above computing device, computer-readable storage medium, and computer program product are of the same concept as the above sample set construction method. For the details not described in detail in the technical solution of the computer program product, reference can be made to the description of the technical solution of the above method. The specific embodiments of this specification have been described above. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims can be executed in a different order than in the embodiments and still achieve the desired results. Additionally, the processes depicted in the drawings do not necessarily require the specific order or sequential order shown to achieve the desired results. In certain embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0213] It should be noted that for the foregoing method embodiments, for the sake of simplicity of description, they are all expressed as a series of action combinations. However, those skilled in the art should know that the embodiments of this specification are not limited by the described order of actions, because according to the embodiments of this specification, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also know that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily essential for the embodiments of this specification. In the above embodiments, the descriptions of each embodiment have their own focuses. For the parts not detailed in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0214] The preferred embodiments of the present specification disclosed above are only used to help explain the present specification. The alternative embodiments do not describe all the details in detail, nor do they limit the invention to the specific embodiments described. Obviously, according to the content of the embodiments of the present specification, many modifications and changes can be made. These embodiments are selected and specifically described in this specification to better explain the principles and practical applications of the embodiments of the present specification, so that those skilled in the art can well understand and utilize this specification. This specification is only limited by the claims and their full scope and equivalents.

Claims

1. A method for constructing a sample set, comprising: Inputting a sample image and preset prompt information into an image annotation model to obtain a global annotation result of the sample image, wherein the global annotation result includes information of a target content type, and the image annotation model is used to extract information of the sample image for global annotation according to the target content type corresponding to the target task based on the prompt of the preset prompt information. The target task includes at least one of tasks such as text description of an image, question answering about an image, and determining corresponding image attributes from a given multiple options for the image. The preset prompt information is constructed based on task requirement information, and the task requirement information indicates the target task to be performed by a model to be trained; Identifying a target object in the sample image, and based on the obtained identification result, marking the target object in the sample image to obtain a global view and intercepting the target object to obtain a local view; Inputting the global view, the local view and visual prompt information into the image annotation model to obtain a local annotation result of the sample image, wherein the visual prompt information is used to prompt the image annotation model to perform an annotation task on the global view based on the local view, the visual prompt information is used to prompt the image annotation model to perform an annotation task, extract information of the local view according to the target content type corresponding to the target task, and determine the association information between the local view and the global view; Constructing a sample set based on the sample image, the global annotation result and the local annotation result.

2. The method according to claim 1, before inputting the global view, the local view and the visual prompt information into the image annotation model, further comprising: Constructing the visual prompt information based on the task requirement information; wherein the task requirement information indicates the target task to be performed by a model to be trained.

3. The method according to any one of claims 1 to 2, the constructing the sample set based on the sample image, the global annotation result and the local annotation result includes: Sending the sample image, the global annotation result and the local annotation result to the front end, and receiving annotation verification information for the sample image sent by the front end; In the case that the annotation verification information indicates that the local annotation result fails the verification and carries region information of the region where the target object is located, intercepting an auxiliary region indicated by the region information in the sample image to obtain an updated local view; Inputting the global view, the updated local view and the visual prompt information into the image annotation model to obtain an updated local annotation result of the sample image; wherein the visual prompt information is used to prompt the image annotation model to perform an annotation task on the global view based on the updated local view; Constructing the sample set based on the sample image, the global annotation result and the updated local annotation result.

4. The method according to any one of claims 1 to 2, the constructing the sample set based on the sample image, the global annotation result and the local annotation result includes: Send the sample image, the global annotation result, and the local annotation result to the front end, and receive the annotation verification information for the sample image sent by the front end; In the case where the annotation verification information indicates that the global annotation result fails the verification and carries a first prompt message, input the sample image and the first prompt message into the image annotation model to obtain an updated global annotation result for the sample image; wherein, the first prompt message is used to prompt the image annotation model to extract information from the input image according to the target content type for global annotation; In the case where the annotation verification information indicates that the local annotation result fails the verification and carries a second prompt message, input the global view, the local view, and the second prompt message into the image annotation model to obtain an updated local annotation result for the sample image; wherein, the second prompt message is used to prompt the image annotation model to perform an annotation task on the global view based on the local view, and the second prompt message is different from the visual prompt message; Construct the sample set based on the sample image, the updated global annotation result, and the updated local annotation result.

5. The method according to any one of claims 1 to 2, wherein constructing a sample set based on the sample image, the global annotation result, and the local annotation result comprises: Fill the global annotation result and the local annotation result into a target prompt message template; wherein, the target prompt message template includes a content part, an instruction part, and an output part, the content part indicates information about the target task to be performed by the question-and-answer model and the scenario it targets, the instruction part indicates the task instruction to be performed by the question-and-answer model, and the global annotation result and the local annotation result are filled into the output part; Generate question-and-answer pairs based on the information in the filled target prompt message template, and construct a sample set for training the question-and-answer model based on the question-and-answer pairs; wherein, the question information in the question-and-answer pairs includes the information in the content part and the instruction part, and the answer information in the question-and-answer pairs includes the information in the output part.

6. The method according to claim 5, before filling the global annotation result and the local annotation result into the target prompt message template, further comprising: Query in multiple prompt message templates for a backup prompt message template whose similarity to the task requirement information is higher than a target threshold; wherein, the task requirement information indicates the target task to be performed by the question-and-answer model; In the case where at least one of the backup prompt message templates is found, use any one of the backup prompt message templates as the target prompt message template.

7. The method according to claim 5, before filling the global annotation result and the local annotation result into the target prompt message template, further comprising: Fill the task requirement information and the information about the scenario it targets into the content part of the target prompt message template, and use a text adjustment model to adjust the content in the target prompt message template.

8. The method according to claim 5, wherein the sample set for constructing and training the Q&A model based on the Q&A pairs includes at least one of the following steps: When the question-and-answer pair meets the complex question-and-answer conditions, disassemble the question-and-answer pair to obtain multiple sub-question-and-answer pairs; construct a sample set for training the question-and-answer model based on the multiple sub-questions; where, The complex Q&A conditions include that the complexity is greater than a target threshold, and / or the format satisfies a set Q&A format; Using a text adjustment model to rewrite the Q&A pair and keep the semantics of the rewritten Q&A pair unchanged; Constructing a sample set for training the Q&A model based on the rewritten Q&A pair; When the answer information in the Q&A pair needs to be inferred from the question information, generating thought chain information on which the answer information is based for the question information; constructing a sample set for training the Q&A model based on the thought chain information and the answer information.

9. The method according to any one of claims 1 to 2, wherein constructing a sample set based on the sample image, the global annotation result, and the local annotation result includes: When the global annotation result and the local annotation result meet the screening conditions, constructing the sample set based on the sample image, the global annotation result, and the local annotation result; wherein the screening conditions include that the content is compliant and / or the corresponding perplexity is within a target range.

10. A method for training a Q&A model, comprising: Obtaining a sample set, wherein the sample set is constructed by the method according to any one of claims 1 to 9; Inputting the sample image in the sample set and the sample question information for the sample image into an initial Q&A model to predict the reference answer information for the sample question information; Training the Q&A model based on the reference answer information, the global annotation result, and the local annotation result of the sample image.

11. A method for Q&A processing, comprising: Obtaining target question information for a target image; Inputting the target image and the target question information into a Q&A model to obtain the answer information for the target question information output by the Q&A model; wherein the Q&A model is trained by the training method according to claim 10.

12. A method for request processing based on a Q&A model, applied to a task platform, comprising: Receiving a model request sent by a terminal device, wherein the model request includes at least one piece of information among a scene identifier of a target scene, scene input data of the target scene, and target model specification parameters; Determining a corresponding Q&A model from at least one model based on the model request; wherein the at least one model is trained by the training method according to claim 10, and the Q&A model is used to obtain the answer information for target question information.

13. The method according to claim 12, wherein the model library stores at least one Q&A model adapted to different scenes and Q&A models with multiple different model specification parameters; determining a corresponding Q&A model from at least one model based on the model request includes: Based on the at least one piece of information in the model request, search for a question-and-answer model in at least one model in the model library that corresponds to the at least one piece of information at the same time; wherein, the scene identifier of the target scene and the scene input data of the target scene correspond to a model adapted to the target scene, and the target model specification parameters correspond to a model with the same model specification parameters as the target model specification parameters. When the at least one piece of information includes the scene input data of the target scene, based on the scene input data of the target scene, train the found question-and-answer model to obtain a trained question-and-answer model.

14. The method according to claim 12 or 13, after determining the corresponding question-and-answer model from at least one model based on the model request, further comprising: Deploy the question-and-answer model, and based on the question-and-answer model, construct a call interface so that the terminal device can call the question-and-answer model through the call interface to obtain a reply message corresponding to the target question information.

15. A task platform, comprising a request interface and a response unit; The said request interface is used to receive a model request sent by a terminal device, where The model request includes at least one of a scene identifier of a target scene, scene input data of the target scene, and model specification parameters; The response unit is configured to determine a corresponding question-and-answer model from at least one model based on the model request, wherein the at least one model is trained by the training method according to claim 10.

16. The task platform according to claim 15, further comprising a call interface, the call interface being constructed based on the question-and-answer model; the call interface is used for the terminal device to call to obtain a reply message of the target question information.

17. A computing device, comprising: A memory and a processor; The memory is used to store computer programs / instructions, and the processor is used to execute the computer programs / instructions. When the computer programs / instructions are executed by the processor, the steps of the method according to any one of claims 1 to 14 are implemented.

18. An electronic device, comprising: A memory and a processor, the memory and the processor are connected by a bus; The memory is used to store computer programs / instructions, and the processor is used to execute the computer programs / instructions. When the computer programs / instructions are executed by the processor, the steps of the method according to any one of claims 1 to 14 are implemented.

19. A computer-readable storage medium, which stores computer programs / instructions, and when the computer programs / instructions are executed by a processor, the steps of the method according to any one of claims 1 to 14 are implemented.

20. A computer program product, comprising computer programs / instructions, and when the computer programs / instructions are executed by a processor, the steps of the method according to any one of claims 1 to 14 are implemented.

Images