Sample set construction method, question and answer model training method, question and answer processing method, request processing method and task platform

Abstract

The embodiment of the invention provides a sample set construction method, a question and answer model training method, a question and answer processing method, a request processing method and a task platform, and the sample set construction method comprises the steps: inputting a sample image into an image annotation model, and obtaining a global annotation result of the sample image; identifying a target object in the sample image, marking the target object in the sample image based on an obtained identification result to obtain a global view, and intercepting the target object to obtain a local view; the global view, the local view and visual prompt information are input into the image annotation model, a local annotation result of the sample image is obtained, and the visual prompt information is used for prompting the image annotation model to execute an annotation task on the global view based on the local view; and constructing a sample set based on the sample image, the global labeling result and the local labeling result. According to the method, the accuracy and the comprehensiveness of the annotation result of the image can be improved.

Description

Technical Field

[0001] The embodiments of this specification relate to the field of computer technology, and in particular to a sample set construction method, a question and answer model training method, a question and answer processing method, a request processing method, and a task platform. Background Art

[0002] With the development of computer technology and artificial intelligence technology, machine learning models have gradually been widely used in various scenarios.

[0003] If a machine learning model is required in a certain scenario, it is necessary to use the training data related to the scenario to train the model, or, for a pre-trained machine learning model, fine-tune the pre-trained machine learning model based on the training data related to the scenario to obtain a machine learning model that can be put into use. The sample data used for model training includes the original data and its labels. The quality of the sample data plays a vital role in the training effect of the model. It is usually necessary to manually annotate the original data, that is, label the original data, to obtain the sample data. For example, for an image analysis model, it is necessary to annotate the sample image, such as annotating the descriptive information of the objects included in the sample image.

[0004] Currently, the effectiveness of data labeling depends on the professionalism of the labelers. The accuracy and comprehensiveness of the labeling results are difficult to guarantee, which will correspondingly lead to insufficient performance of the trained model. Summary of the invention

[0005] The embodiments of this specification provide a sample set construction method, which can improve the accuracy and comprehensiveness of the annotation results for images, and correspondingly improve the performance of the model trained based on the sample set. One or more embodiments of this specification also involve a question-answering model training method, a question-answering processing method, a request processing method based on the question-answering model, a task platform, a computing device, an electronic device, a computer-readable storage medium, and a computer program product.

[0006] The sample set construction method includes: Inputting the sample image into the image annotation model to obtain a global annotation result of the sample image; Identify the target object in the sample image, and based on the obtained recognition result, mark the target object in the sample image to obtain a global view, and intercept the target object to obtain a local view; Inputting the global view, the local view and the visual cue information into the image annotation model to obtain a local annotation result of the sample image, wherein the visual cue information is used to prompt the image annotation model to perform an annotation task on the global view based on the local view; A sample set is constructed based on the sample image, the global annotation result and the local annotation result.

[0007] In one embodiment of the present specification, on the basis of annotating the global view of the sample image, object recognition and corresponding local view capture are performed on the sample image, and the image annotation model is instructed to locally annotate the image using visual cue information for the global view and the local view, and a sample set is constructed based on the global annotation results and the local annotation results. In this way, a more comprehensive and complete annotation of the sample image can be ensured, and the accuracy and comprehensiveness of the annotation results can be improved. Accordingly, the sample set obtained based on the annotation results can achieve a better training effect on the model, ensuring a model with better performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] Figure 1 It is a structural diagram of a question-answering system provided in an embodiment of this specification; Figure 2 is a flow chart of a sample set construction method provided in an embodiment of this specification; Figure 3 This is an overall implementation framework diagram of a sample set construction method provided in an embodiment of this specification; Figure 4 is a flow chart of a method for training a question-answering model provided in an embodiment of this specification; Figure 5 is a flow chart of a question-answering processing method provided in an embodiment of this specification; Figure 6 is a flowchart of a request processing method based on a question-answer model provided in an embodiment of this specification; Figure 7 It is a structural diagram of a task platform provided in an embodiment of this specification; Figure 8 is a schematic diagram of the structure of a sample set construction device provided in an embodiment of this specification; Fig. 9 It is a structural schematic diagram of a training device for a question-answering model provided in an embodiment of this specification; Fig.10 It is a structural diagram of a question-answering processing device provided in an embodiment of this specification; Fig.11 is a structural block diagram of a computing device provided in an embodiment of this specification; Fig.12 It is a structural block diagram of an electronic device provided in one embodiment of this specification. DETAILED DESCRIPTION

[0009] Many specific details are described in the following description to facilitate a full understanding of this specification. However, this specification can be implemented in many other ways than those described herein, and those skilled in the art can make similar generalizations without violating the connotation of this specification, so this specification is not limited to the specific implementation disclosed below.

[0010] The terms used in one or more embodiments of this specification are only for the purpose of describing specific embodiments, and are not intended to limit one or more embodiments of this specification. The singular forms of "a" and "the" used in one or more embodiments of this specification and the appended claims are also intended to include plural forms, unless the context clearly indicates other meanings. It should also be understood that the term "and / or" used in one or more embodiments of this specification refers to and includes any or all possible combinations of one or more associated listed items.

[0011] It should be understood that although the terms first, second, etc. may be used to describe various information in one or more embodiments of this specification, this information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of one or more embodiments of this specification, the first may also be referred to as the second, and similarly, the second may also be referred to as the first. Depending on the context, the word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining".

[0012] In addition, it should be noted that the user information (including but not limited to user device information, user personal information, user behavior information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in one or more embodiments of this specification are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0013] In one or more embodiments of this specification, a large model refers to a deep learning model with large-scale model parameters, which usually contains hundreds of millions, tens of billions, hundreds of billions, trillions, or even more than 10 trillion model parameters. A large model can also be called a foundation model / foundation model. The large model is pre-trained with large-scale unlabeled corpus to produce a pre-trained model with more than 100 million parameters. This model can adapt to a wide range of downstream tasks, and the model has good generalization ability, such as a large-scale language model (LLM, Large Language Model), a multi-modal pre-training model, etc.

[0014] When the large model is actually applied, only a small number of samples are needed to fine-tune the pre-trained model and it can be applied to different tasks. The large model can be widely used in natural language processing (NLP), computer vision and other fields, and can be specifically applied to computer vision tasks such as visual question answering (VQA), image description (IC, Image Caption), image generation, and text-based sentiment classification, text data annotation, machine translation and other natural language processing tasks. The main application scenarios of the large model include digital assistants, intelligent robots, search, online education, office software, e-commerce, intelligent design, etc. The models involved in the embodiments of this specification include but are not limited to the above-mentioned large models, and can be any type of machine learning model. For example, it can be a BERT model, a convolutional neural network (CNN) model, a Transformer model, etc.

[0015] The model involved in the embodiments of this specification can be a large model or other smaller models, which is not limited in the embodiments of this specification.

[0016] With the development of computer technology and artificial intelligence technology, machine learning models are gradually used in various fields to assist in the execution of tasks. The requirements for the performance of machine learning models are getting higher and higher. The performance of machine learning models depends to a great extent on the quality of the sample data used for training. If high-quality sample data is used to train the machine learning model, the performance of the trained model will be higher. For models used in different fields, if the data related to the field is used for model training, the application effect of the trained model in the field will also be better. Therefore, the construction effect of the sample data required for model training is crucial.

[0017] In recent years, large models have gradually evolved from single-modality to general artificial intelligence with multi-modal fusion. Large models launched by different companies have exploded and have shown strong capabilities in general fields such as visual question answering (VQA) and image description. However, for industry fields that contain more professional knowledge, the application of large models still faces many challenges in actual use. It is usually necessary to build a richer and more professional sample set (such as industry graphic and text datasets) for the large model in each field to fine-tune the model to enhance its industry professional capabilities.

[0018] The annotation and dataset construction methods for graphic data are still in a relatively primitive state in the traditional visual model era, that is, it is necessary to manually frame the target object in the picture or manually describe the content in the picture, which consumes a lot of annotation manpower costs. At the same time, for more professional industry fields, the annotation results are very dependent on the professionalism of the annotators. The quality of the annotation results given by annotators with low professionalism may not meet the requirements. Annotators usually annotate based on a given fixed template, which will lead to low data quality problems such as severe templateization of annotation results and high content error rate. For pictures containing many entities, it is usually difficult for annotators to give an accurate and comprehensive graphic description. For the annotated data, it is usually difficult to obtain the model capabilities that meet the requirements when directly used for training. Data processing links such as data cleaning, industry content supplementation, and data evolution enhancement are also required.

[0019] The embodiments of this specification provide a sample set construction method, which can accurately and comprehensively annotate sample images, construct a high-quality sample set, and accordingly improve the performance of the model trained based on the sample set. One or more embodiments of this specification also involve a question-answering model training method, a question-answering processing method, a request processing method based on a question-answering model, a task platform, a computing device, an electronic device, a computer-readable storage medium, and a computer program product.

[0020] In the embodiments of this specification, the sample set constructed using the sample set construction method can be used to train a question-answering model, and the question-answering model is used to implement the question-answering processing method. Figure 1 is a schematic diagram of the structure of a question-answering system provided in an embodiment of this specification. The question-answering processing method provided in an embodiment of this specification can be applied to the question-answering system. Figure 1 As shown, the question-answering system may include a cloud platform 10 and a terminal device 20 that are communicatively connected.

[0021] The terminal device 20 is used to send information about a problem to be solved to the cloud platform 10. The cloud platform 10 is used to execute the question-and-answer processing method provided in the embodiment of this specification based on the question information, obtain the reply information corresponding to the question information, and send the reply information to the terminal device 20. The cloud platform 10 can be a background server for some applications or web pages with question-and-answer functions, which is used to support the normal operation of the application. The terminal device 20 can be installed with the application, and the terminal device 20 can achieve communication connection with the cloud platform 10 by running the application, and interact with the cloud platform 10 to complete the question-and-answer task. In some embodiments, in addition to providing the question-and-answer function, the application can also provide other functions, such as social communication function, video playback function, image processing function, etc.

[0022] In such Figure 1 In the application scenario shown, the question-answering model (such as a large model) used in the implementation process of the question-answering processing method is deployed in the cloud platform 10, and the cloud platform 10 can be connected to one or more terminal devices 20 through a local area network connection, a wide area network connection, an Internet connection, or other types of data networks. The terminal device 20 here may include but is not limited to: a smart phone, a tablet computer, a laptop computer, a PDA, a personal computer, a smart home device, a vehicle-mounted device, etc. The terminal device 20 can interact with the user through a graphical user interface to implement the call of the question-answering model, and then implement the question-answering processing method provided in the embodiment of this specification. If the computing resources of the terminal device 20 are large enough, the large model can also be directly deployed on the terminal device, and the terminal device directly uses the large model to implement the question-answering processing method provided in the embodiment of this specification. The question-answering model can also be trained on the cloud platform 10, and accordingly, the training method of the question-answering model provided in this specification can also be executed by the cloud platform 10.

[0023] The sample set construction method provided in the embodiments of this specification may also be executed by the cloud platform 10, and the image annotation model used in the sample set construction method may be deployed in the cloud platform 10. In some implementations, the sample set construction method may also be executed by the terminal device 20, such as the terminal device 20 calling the image annotation model deployed on the cloud platform 10 to execute the sample set construction method, or the image annotation model may be deployed in the terminal device 20.

[0024] There is also a certain feedback mechanism between the terminal device 20 and the cloud platform 10. The terminal device 20 can also send feedback information for the reply information to the cloud platform 10, and the cloud platform 10 can adjust the question-answering model based on the feedback information. The cloud platform 10 can also update the sample set used in the training process of the question-answering model based on the feedback information, so that the terminal device 20 can continue to train and adjust the question-answering model based on the updated sample set.

[0025] Figure 2 is a flow chart of a sample set construction method provided in an embodiment of the present specification. The following is an example of the sample set construction method being executed by a construction device. In some embodiments, the method can be applied to Figure 1 The cloud platform 10 in the embodiment, for example, the construction device may be the cloud platform 10. Figure 2 As shown, the method may include the following steps 202 to 208 .

[0026] Step 202: Input the sample image into the image annotation model to obtain a global annotation result of the sample image.

[0027] The image annotation model in the embodiments of this specification may be an existing trained image annotation model, and the image annotation model may be a vision-language model (VLM), which is an artificial intelligence model that combines visual information and language information.

[0028] The sample images may be images collected by staff, or images collected from the Internet using computing equipment, or images provided by customers, and are not limited here. In some embodiments, the sample images may be related to the application field and scenario of the model to be trained. For example, the constructed sample set is used to train picture-text question-and-answer scenarios applied to the field of natural scenery. Accordingly, the sample images used to construct the sample set may be more natural scenery images. For another example, the constructed sample set is used to train question-and-answer scenarios applied to the field of celestial bodies and astronomical phenomena. Accordingly, the sample images used to construct the sample set may be more space star images.

[0029] The construction device may input the sample image into the image annotation model, so that the image annotation model analyzes and annotates the sample image according to its own originally set annotation logic, and obtains a global annotation result. The global annotation result may be description information for the sample image. In an example, the description information includes information such as the general content contained in the sample image, the field to which the sample image belongs, and the overall style of the sample image.

[0030] In some embodiments, the model to which the constructed sample set is to be applied can be predetermined, and task requirement information for the model can be obtained. The task requirement information indicates the requirements for the model, such as indicating the field and scenario to which the model is applied, the tasks to be performed by the model, and the task execution logic of the model. The construction device can annotate the sample images accordingly based on the task requirement information to ensure that the obtained annotation results match the model to be trained, and to ensure that a better model training result is obtained based on the annotation results.

[0031] To call the image annotation model to perform the annotation task, it is necessary to input prompt information (prompt) to the image annotation model to prompt the model to perform the corresponding processing and output the results that meet the requirements. Prompt refers to a prompt word, which is a natural language input, similar to a command or instruction, used to instruct the model what actions to take or what output to generate when performing a specific task. In the embodiments of this specification, corresponding prompt information can be constructed based on the task requirement information to instruct the image annotation model to output a global annotation result that meets the requirements.

[0032] Accordingly, before the above step 202, the sample set construction method provided in the embodiment of the present specification further includes: constructing preset prompt information based on the task requirement information; wherein the task requirement information indicates the target task to be performed by the model to be trained; the preset prompt information is used to prompt the image annotation model to extract the information of the input image for global annotation according to the target content type corresponding to the target task. In step 202, the sample image is input into the image annotation model to obtain the global annotation result of the sample image, including: inputting the sample image and the preset prompt information into the image annotation model, so that the image annotation model extracts the information of the sample image according to the target content type corresponding to the target task, and obtains the global annotation result of the sample image; wherein the global annotation result includes the information of the target content type.

[0033] For different models, such as models applied to different scenarios or realizing different functions, the corresponding task requirement information may be different. The task requirement information at least indicates the target task that the model is used to perform. A model can perform one or more target tasks, and the number of target tasks indicated by the task requirement information can be one or more. For example, the target tasks that the question-answering model can perform for an image may include: text description of the image, answering questions for the image, or determining corresponding image attributes for the image from multiple given options. The target tasks may also include other tasks, which are not limited here.

[0034] For different target tasks, different types of content can be annotated for images to achieve the target task based on the obtained annotated content training model. Each target task corresponds to a target content type to be annotated. For example, for the target task of textual description of images, the corresponding target content type is the textual description information of the image. For the target task of answering questions for images, the corresponding target content type is the answer information of the given sample question determined for the image. For the target task of determining image attributes for images, the corresponding target content type is the image attributes to which the image belongs.

[0035] The construction device constructs preset prompt information based on the target content type corresponding to at least one target task indicated by the task requirement information. The preset prompt information is used in the global annotation process of the input sample image, prompting the image annotation model to analyze the sample image and extract information to obtain the information of the target content type, and use the information of the target content type as the global annotation result of the sample image. The construction device inputs the sample image and the preset prompt information into the image annotation model, so that the image annotation model processes based on the prompt of the preset prompt information.

[0036] In some embodiments, for the model to be trained, the construction device may construct preset prompt information only once based on the task requirement information corresponding to the model, and the preset prompt information may be applicable to all sample images. The same preset prompt information may be used for each sample image to obtain a global annotation result for the sample image, without the need to construct new prompt information for different sample images.

[0037] In the embodiments of the present specification, the image annotation model is instructed to implement global annotation of the image based on preset prompt information. The preset prompt information is constructed based on the task requirement information of the model to be trained. This ensures that the obtained annotation results can better match the model, ensures that the obtained annotation results are more suitable for model training, and ensures that the trained model has better performance.

[0038] Step 204: Identify the target object in the sample image, and based on the obtained identification result, mark the target object in the sample image to obtain a global view, and capture the target object to obtain a local view.

[0039] In the embodiments of this specification, in addition to global annotation of the sample image, the construction device also performs local annotation on the sample image to improve the completeness and comprehensiveness of the annotation of the sample image. For local annotation, the construction device first needs to determine the local information in the sample image (that is, the information located in the local area), and then analyze the local information to achieve local annotation. The local annotation can be an annotation for certain specific objects in the sample image. For example, the construction device can perform object recognition on the sample image to identify the target object in the sample image, and obtain local information of the target object in the sample image based on the target object. The local information includes information about the target object, such as the category to which the target object belongs, the current state of the target object, and the relationship between the target object and surrounding objects. The target object can also be called a detail entity in the image.

[0040] In one implementation, the local information in the sample image includes a local view in the sample image, such as a local view where the target object is located. The construction device can use techniques such as image detection and segmentation, such as using visual models such as image detection and segmentation to determine the local view of the sample image that contains the target object, and perform local annotation based on the local view. For example, the construction device can intercept the target object identified in the sample image to obtain a local view. In the process of local annotation, the construction device can also combine the global view of the sample image so that the local annotation result obtained includes the association information between the information in the local view and the information in the global view, to ensure that more complete and detailed information is obtained, and to ensure that the annotation completeness of the sample image is high. For example, the global view can be a view obtained by marking the target object based on the sample image.

[0041] For different models, the categories of target objects that need to be determined are different. For example, for an image that contains both people and plants, the person behavior analysis model focuses on the person. If the person behavior analysis model is trained, the target object determined during labeling should be the person, thereby obtaining the person's information; the plant recognition model focuses on the plant. If the plant recognition model is trained, the target object determined during labeling should be the plant, thereby obtaining the plant's information. In some embodiments, the construction device can determine the required object type based on the task requirement information, and then perform object recognition on the sample image based on the object type to determine the target object of the object type. This ensures that the local annotation based on the target object matches the model requirements, and ensures that the model training effect based on the local annotation results is better. The object type determined based on the task requirement information can be one or more, which is not limited here.

[0042] The construction device can identify the target object in the sample image, and the recognition result can characterize the area where each target object is located. For example, the recognition result can include a detection frame or an enclosing polygon of each target object, and the detection frame and the enclosing polygon indicate the coordinates of the minimum area where the target object is located. The construction device can directly intercept the detection frame or the enclosing polygon to obtain a local view. In some embodiments, based on the minimum area where the target object is located, the construction device can adjust the area to intercept a larger area containing the target object as a local view. In this way, more detailed local information can be obtained based on the local view, such as in addition to obtaining the information of the target object itself, the association relationship between the target object and its background can also be obtained, ensuring that the local annotation result is more complete and detailed. For example, the construction device can expand the area range based on the minimum area and the set expansion parameter to obtain the target area containing the target object, and then intercept the target area in the sample image to obtain a local view. The expansion parameter is greater than 1, such as the expansion parameter can be 1.2, 1.1 or other values.

[0043] In one implementation, if an enclosing polygon is identified for the target object, the construction device can construct a circumscribed rectangular frame of the polygon based on the point with the largest coordinate value and the point with the smallest coordinate value in the enclosing polygon. For example, a rectangular coordinate system is constructed with the lower left vertex of the sample image as the origin, and the lower leftmost point and the upper rightmost point in the enclosing polygon can be used as the two diagonal points of the rectangle to construct a circumscribed rectangular frame. For the circumscribed rectangular frame or detection frame of the target object, the coordinates of the target area are calculated by magnifying its center by a fixed multiple (such as the above-mentioned expansion parameter). For example, the coordinates of the detection frame are expressed as B=(x, y, w, h), where x and y represent the horizontal and vertical coordinates of the upper left point of the detection frame, respectively, w represents the width of the detection frame, and h represents the height of the detection frame. The coordinates of the target area can be B'=scale(B, c)=(x+w / 2-c×w / 2, x+h / 2-c×h / 2, c×w, c×h), where c represents the expansion parameter. Then, the target area can be cut out from the sample image based on the coordinates of the target area to obtain a local view. The construction device can also draw a marking box (such as a red rectangular box) at a corresponding position on the sample image based on the coordinates of the target area, and save it as a global view of the sample image.

[0044] In some implementations, if multiple target objects are identified in the sample image, a corresponding global view and a local view may be determined for each target object.

[0045] Step 206: input the global view, the local view and the visual cue information into the image annotation model to obtain the local annotation result of the sample image, wherein the visual cue information is used to prompt the image annotation model to perform the annotation task on the global view based on the local view.

[0046] The construction device can continue to use the image annotation model to perform local annotation on the sample image. In the process of local annotation, the global view and local view of the sample image determined for the target object can be combined. For the image annotation model, reference can be made to the relevant introduction in the aforementioned step 202, which will not be repeated here.

[0047] The construction device may first construct visual prompt information to instruct the image annotation model to perform the annotation task on the global view based on the local view, so as to realize the local annotation of the sample image. Accordingly, before inputting the global view, the local view and the visual prompt information into the image annotation model in step 206, the sample set construction method provided in the embodiment of the present specification also includes: constructing visual prompt information based on the task requirement information; wherein the task requirement information indicates the target task to be performed by the model to be trained; the visual prompt information is used to prompt the image annotation model to perform the annotation task, extract information from the input local view according to the target content type corresponding to the target task, and determine the association information between the input local view and the global view. The visual prompt information includes a multi-image input instruction, indicating that the input global view and local view are to be analyzed.

[0048] In the present embodiment, the construction device can construct the visual prompt information based on the task requirement information. For the task requirement information, the target task and its corresponding target content type, reference can be made to the relevant introduction in the aforementioned step 202, which will not be repeated here. The visual prompt information can instruct the image annotation model to extract information for the input partial view. For example, the information of the target content type in the partial view is extracted, such as information related to the target object, and the association information between the target object in the partial view and other external content (such as the background of the target object, the scene in which it is located, etc.) can also be included. The visual prompt information also instructs the image annotation model to determine the association information between the partial view and the global view for the input partial view and the global view, so as to describe the partial view based on the association information and the information extracted from the partial view (such as information including the target object itself). For example, the sample image is an image of an amusement park, and the target object in the partial view is a child. The local annotation result obtained can be that the sample image contains a child playing in the amusement park, and the child also interacts with the puppy next to him.

[0049] In the embodiments of the present specification, the visual prompt information prompts the image annotation model to annotate the association information between the local view and the global view during the local annotation process. This ensures that the local information in the image can be more comprehensively described through local annotation, thereby improving the effect of model training based on the local annotation results.

[0050] Step 208: construct a sample set based on the sample image, the global annotation result and the local annotation result.

[0051] The construction device can determine a sample data for each sample image. The aforementioned global labeling and local labeling process is performed on multiple sample images to obtain multiple sample data, and then the multiple sample data can form a sample set. Each sample data includes a sample image, and a global labeling result and a local labeling result of the sample image. For a sample image, if multiple target objects are detected for it, a local labeling result can be obtained for each target object. All local labeling results of the sample image can belong to one sample data.

[0052] In one embodiment, after obtaining the global annotation result and the local annotation result for a sample image, the construction device can directly determine the sample image, the global annotation result and the local annotation result as a sample data. In another embodiment, the construction device can further analyze or process the obtained global annotation result and the local annotation result to construct a sample set based on the data obtained after the analysis or processing and the original sample image.

[0053] In the embodiment of the present specification, after the sample image is annotated, the annotation result can be verified. After the verification is passed, a sample set is constructed based on the obtained annotation result to ensure the accuracy and perfection of the annotation result. Accordingly, the above step 208 may include: when the global annotation result and the local annotation result of the sample image are both verified, a sample set is constructed based on the sample image, the global annotation result and the local annotation result. In the case that the verification fails, the sample image can be re-annotated to construct a sample set based on the re-annotated result. For example, the verification process can be manually verified using an annotation tool, and the verification personnel check the global annotation result and the local annotation result, check the generated annotation result in the annotation tool, determine whether the annotation result is accurate and perfect, and modify the unannotated content or erroneous content. In the embodiment of the present specification, the annotation process before the verification can be called a pre-annotation process, and the obtained annotation result can be called a pre-annotation result. After the pre-annotation result is verified, the final annotation result can be determined.

[0054] In one implementation, constructing a sample set based on the sample image, the global annotation result, and the local annotation result in step 208 includes: Send the sample image, global annotation results and local annotation results to the front end, and receive the annotation verification information for the sample image sent by the front end; When the annotation verification information indicates that the local annotation result has not passed the verification and carries the region information of the region where the target object is located, the auxiliary region indicated by the region information of the sample image is intercepted to obtain an updated local view; The global view, the updated local view and the visual cue information are input into the image annotation model to obtain the newly added local annotation result of the sample image; wherein the visual cue information is used to prompt the image annotation model to perform the annotation task on the global view based on the updated local view; A sample set is constructed based on sample images, global annotation results, and newly added local annotation results.

[0055] In an embodiment of the present specification, after the construction device generates corresponding global annotation results and local annotation results for the sample image, these annotation results can be sent to the front end for verification by the verification personnel, and the front end refers to the terminal device used by the verification personnel. The terminal device uses the annotation tools installed therein to display the received sample image, global annotation results and local annotation results for verification by the verification personnel. When it is determined that the global annotation results and local annotation results are accurate and complete, the verification personnel can trigger the terminal device to send annotation verification information to the construction device, and the annotation verification information indicates that the annotation verification for the sample image has passed. For example, a control for confirming that the passing can be displayed on the page for verifying the annotation results on the terminal device, and the verification personnel can click on the control to trigger the corresponding annotation verification information. Based on the annotation verification information, the construction device can generate a sample data based on the sample image and its global annotation results and local annotation results, and store it in the sample set.

[0056] When the verification personnel determines that there are errors or incomplete annotations in the global annotation results or local annotation results, the annotation content can be modified and supplemented through the terminal device. The page for verifying the annotation results on the terminal device can display controls for modifying the annotation content and controls for adding annotation content. The verification personnel can click the corresponding controls and perform operations on the sample image on the page (such as clicking, drawing frames or polygonal box selection, etc.) to modify or add annotation content. After the operation, the verification personnel can trigger the device on the terminal to generate annotation verification information and send it to the construction device. For example, the annotation verification information can indicate that the annotation result has not passed the verification. In some embodiments, the verification personnel can also specifically choose to modify and supplement the global annotation results, or modify or supplement the local annotation results. Accordingly, the generated annotation verification information indicates that the global or local annotation results have not passed the verification.

[0057] In the case where the local annotation result fails to pass the verification, it is possible that the recognition of the target object is incomplete or incorrect. The verification personnel can mark points or draw a rectangular box for the target object in the sample image to input the regional information of the area where the target object is located. Accordingly, the annotation verification information carries the regional information of the area where the target object is located. The construction device can intercept the auxiliary area indicated by the regional information in the sample image based on the regional information (such as calling the segmentation model to obtain the polygonal segmentation result of the target object), obtain an updated local view, and then re-execute the local annotation process in the above step 206 based on the updated local view to obtain an updated local annotation result. After that, a sample set can be constructed based on the updated local annotation result, such as determining the sample image and its global annotation result and the updated local annotation result as a sample data. The updated local annotation result can also be further verified by the verification personnel, and then used to construct the sample set after the verification passes.

[0058] If the verification personnel determine that a certain local annotation result is wrong, the local annotation result can be deleted, such as making the annotation verification information carry a deletion instruction for the local annotation result. Correspondingly, the construction device can delete the local annotation result based on the annotation verification information. Through such deletion and the aforementioned determination and update of the local annotation result, the local annotation result can be corrected and improved.

[0059] In one embodiment, there may be multiple target objects in a sample image. By locally annotating the sample image, multiple local annotation results corresponding to the multiple target objects can be obtained. During the verification process, the multiple local annotation results can be verified respectively, and the annotation verification information can indicate that the local annotation result corresponding to any target object has not passed the verification, and carries the area information for the target object. Accordingly, the updated local annotation result obtained is also for the target object. At this time, the updated local annotation result can be used to directly replace the original local annotation result of the target object, without changing the original local annotation results of other target objects.

[0060] Optionally, the target object for the annotation verification information may also be an object that has not been partially annotated originally. In this case, the updated local annotation results may be added to the original local annotation results to obtain the final local annotation result for the sample image.

[0061] In some embodiments, at least one annotation auxiliary plug-in may be installed in the terminal device to provide different annotation functions, assist the verification personnel in verification, and improve labor efficiency. The determination of the area where the target object is located can be implemented based on the annotation auxiliary plug-in of the interactive target segmentation tool. If the plug-in can be installed in the terminal device, the verification personnel can start the plug-in during the verification of the annotation results.

[0062] In one implementation, constructing a sample set based on the sample image, the global annotation result, and the local annotation result in step 208 includes: Send the sample image, global annotation results and local annotation results to the front end, and receive the annotation verification information for the sample image sent by the front end; When the annotation verification information indicates that the global annotation result has not passed the verification and carries the first prompt information, the sample image and the first prompt information are input into the image annotation model to obtain an updated global annotation result of the sample image; wherein the first prompt information is used to prompt the image annotation model to extract information of the input image for global annotation according to the target content type; When the annotation verification information indicates that the local annotation result has not passed the verification and carries the second prompt information, the global view, the local view and the second prompt information are input into the image annotation model to obtain an updated local annotation result of the sample image; wherein the second prompt information is used to prompt the image annotation model to perform the annotation task on the global view based on the local view, and the second prompt information is different from the visual prompt information; Based on the sample images, the global annotation results are updated and the local annotation results are updated to construct a sample set.

[0063] Regarding the marking verification information, please refer to the relevant introduction in the above-mentioned previous implementation method, which will not be repeated here.

[0064] In the embodiments of the present specification, when the global annotation result or the local annotation result fails to pass the verification, it is possible that the prompt information used in the process of performing the corresponding annotation using the image annotation model is not accurate enough. The verification personnel can construct more accurate prompt information by themselves, and use the prompt information to instruct the image annotation model to re-execute the global annotation or local annotation to obtain more accurate global annotation results and local annotation results. The prompt information reconstructed by the verification personnel can be carried in the standard verification information and sent to the construction device, and the construction device re-enters the prompt information into the image annotation model to instruct the image annotation model to re-execute the aforementioned step 202 or step 206 to re-perform the global annotation or local annotation to obtain an updated annotation result, and then construct a sample set based on the updated annotation result. The updated annotation result can also be further verified by the verification personnel, and then used to construct the sample set after the verification passes.

[0065] In the embodiments of this specification, the prompt information reconstructed for the task of global annotation is called the first prompt information. The first prompt information can be analogous to the aforementioned preset prompt information. The first prompt information is also used to prompt the image annotation model to extract the information of the input image for global annotation according to the target content type corresponding to the target task. The first prompt information is different from the aforementioned preset prompt information, such as different specific semantics or different expression forms. The first prompt information can be modified by the verification personnel based on the preset prompt information, or it can also be regenerated. In the case where the annotation verification information carries the first prompt information, the sample image and the first prompt information are input into the image annotation model to obtain an updated global annotation result of the sample image.

[0066] The visual prompt information reconstructed for the task of local annotation is called the second prompt information. The second prompt information can be analogous to the aforementioned visual prompt information. The second prompt information is also used to prompt the image annotation model to perform the annotation task, and extract the information of the local view and its association with the global view according to the target content type corresponding to the target task. The second prompt information is different from the specific content of the visual prompt information, such as different specific semantics or different expressions. The second prompt information can be modified by the verification personnel based on the aforementioned visual prompt information, or it can be regenerated. When the annotation verification information carries the second prompt information, the global view, the local view and the second prompt information are input into the image annotation model to obtain an updated local annotation result of the sample image.

[0067] In one implementation, the first prompt information and the second prompt information can be implemented by generating the annotation auxiliary plug-in based on the auxiliary description installed in the terminal device. For example, the plug-in can be installed in the terminal device, and the verification personnel can start the plug-in during the verification of the annotation results.

[0068] In an optional manner, an optical character recognition (OCR) plug-in may also be installed in the terminal device, and the verification personnel may interactively mark the text area in the sample image and draw a rectangular box. The coordinates of the rectangular box may then be input into the OCR plug-in to output the corresponding text content, and corresponding annotation results may be added based on the text content. A sample set may be constructed based on the annotation results combined with the aforementioned global annotation results and local annotation results. For example, the text content may be directly used as a newly added local annotation result, and the local annotation result may be merged with the originally obtained local annotation result to construct a sample set. In an optional manner, certain adjustments are made to the originally obtained global annotation results and local annotation results based on the text content, and a sample set is constructed based on the adjusted annotation results.

[0069] The following takes the question-answering model as an example to introduce the specific construction method of the sample set. In the embodiment of this specification, the construction device can construct question-answering pairs based on the obtained annotation results based on a preset prompt information template, and use the obtained question-answering pairs as sample data used for training the question-answering model. These prompt information templates can be standard prompt information templates used historically in the industry, or they can be prompt information templates generated based on industry needs using a model.

[0070] In some implementations, constructing a sample set based on the sample image, the global annotation result, and the local annotation result in step 208 includes: Fill the global annotation results and the local annotation results into the target prompt information template; wherein the target prompt information template includes a content part, an instruction part and an output part, the content part indicates the target task to be performed by the question-answering model and the information of the scene it targets, the instruction part indicates the task instruction to be performed by the question-answering model, and the global annotation results and the local annotation results are filled into the output part; Based on the information in the filled target prompt information template, a question-answer pair is generated, and a sample set used for training the question-answer model is constructed based on the question-answer pair; wherein the question information in the question-answer pair includes information of the content part and the instruction part, and the answer information in the question-answer pair includes information of the output part.

[0071] In an embodiment of the present specification, each prompt information template to be selected may include a content part (context), an instruction part (instruction) and an output part (output), wherein the content part indicates the target task for the question-answering model to be executed and the information of the scene for which it is targeted, the instruction part indicates the task instruction to be executed by the question-answering model, and the global annotation result and the local annotation result are filled in the output part. Keywords corresponding to the industry or task will be established in each template. For example, a prompt information template includes context: "This is a {scenario category} scene, focusing on the following events: {task requirement information}", instruction: "Are there any obvious abnormal events in this scene?", output: "I found the following abnormal events: {annotation content}". The "scenario category" is used to fill in the information of the scene targeted by the question-answering model, the "task requirement information" is used to fill in the requirement information related to the target task for which the question-answering model is used to execute, and the "annotation content" is used to fill in the global annotation results and local annotation results obtained by annotating the sample image. The scene category and task requirement information can be given by the user.

[0072] After obtaining the annotation results for the sample image, the construction device can fill the annotation results into the selected target prompt template, and automatically assemble and organize them into a question-answer pair based on the information in the template. For example, the format is {Q: Context+Instruction; A: Ouput}, where Q represents question information and A represents answer information.

[0073] In some embodiments, a sample image is annotated to generate multiple local annotation results. The construction device can summarize the multiple local annotation results, such as by summarizing them through a large model, and generate a more rich and comprehensive overall description information, and then generate a question-answer pair based on the overall description information to ensure the richness and comprehensiveness of the response information.

[0074] In the embodiment of this specification, before using the prompt information template to construct a question-answer pair, it is necessary to first determine the target prompt information template to be used for the sample image. In some embodiments, before filling the global annotation results and the local annotation results into the target prompt information template, the sample set construction method provided in the embodiment of this specification also includes: among multiple prompt information templates, querying a spare prompt information template whose similarity with the task requirement information is higher than the target threshold; wherein the task requirement information indicates the target task for the question-answering model to perform; when at least one spare prompt information template is queried, any spare prompt information template is used as the target prompt information template.

[0075] The construction device can determine the target prompt information template based on the task requirement information. Among the multiple prompt information templates to be selected, each prompt information template can correspond to an applicable field, a scene, and a task, such as the "scene category" and "task requirement information" in the content part of each prompt information template are pre-filled with corresponding information. The construction device can compare the task requirement information corresponding to the model to be trained with the information in each prompt information template to determine the similarity. When the similarity is higher than the set target threshold, the corresponding prompt information template is used as a backup prompt information template. In one method, the construction device compares the scene to be applied to the model to be trained and the corresponding task requirement information with the scene and task requirement information corresponding to each prompt information template to determine the similarity. For the model to be trained, when using each sample image for annotation and constructing a question-answer pair, the target prompt information template can be selected from the backup prompt information templates determined here, without determining the corresponding alternative prompt information template for each sample image.

[0076] If only one backup prompt template is determined, the backup prompt information template is directly determined as the target prompt information template corresponding to each sample image. In the case where multiple backup prompt templates are determined, any backup prompt template can be randomly selected from the multiple backup prompt templates as the target prompt information template for each sample image. In this way, it can be ensured that the form or expression of the question-answer pairs constructed for different sample images is different, the flexibility of the construction of the question-answer pairs is guaranteed, the rigid expression form is avoided, and sample data with more expressions can be obtained, ensuring a better training effect on the model.

[0077] In some embodiments, before filling the global annotation results and the local annotation results into the target prompt information template, the sample set construction method provided in the embodiment of the present specification also includes: filling the task requirement information and the information of the scene it targets into the content part of the target prompt information template, and adjusting the content in the target prompt information template using the text adjustment model.

[0078] The construction device can obtain the task requirement information corresponding to the model to be trained and the information of the scene it targets, and fill these information into the "task requirement information" and "scene category" in the target prompt information template. For example, the original information in the "task requirement information" and "scene category" in the target prompt information template can be replaced with the task requirement information corresponding to the model to be trained and the information of the scene it targets. In some embodiments, the construction device can also enhance the target prompt template to ensure that the filled template information is more sufficient and rich. For example, the construction device can obtain industry background knowledge for the model to be trained, and also fill the industry background knowledge into the content part of the template. For example, add a "background knowledge" part after the "scene category", and fill the acquired industry background knowledge into this position.

[0079] The construction device can also enhance the target prompt template using the text adjustment model to improve the fluency and completeness of the template information, and can rewrite the template information slightly to make the rewritten information more consistent with the filled scene category and task requirement information. The text adjustment model can be a large language model.

[0080] In some implementations, the sample set used to construct the training question-answer model based on the question-answer pairs includes at least one of the following steps: When the question-answer pair meets the complex question-answer condition, the question-answer pair is disassembled to obtain multiple sub-question-answer pairs; a sample set used for training the question-answer model is constructed based on the multiple sub-questions and answers; wherein the complex question-answer condition includes that the complexity is greater than a target threshold, and / or the format satisfies a set question-answer format; Using the text adjustment model, the question-answer pair is rewritten, and the semantics of the rewritten question-answer pair remains unchanged; based on the rewritten question-answer pair, a sample set used for training the question-answer model is constructed; When the answer information in a question-answer pair needs to be inferred based on the question information, thought chain information based on which the answer information is derived is generated based on the question information; based on the thought chain information and the answer information, a sample set used to train the question-answering model is constructed.

[0081] In one method, the construction device can decompose the generated question-answer pairs. For some question-answer pairs that meet complex question-answer conditions (such as long descriptions and complex content), the large model can be used to decompose them into multiple sub-question-answer pairs. For example, based on the description of the original question-answer pair, the large model automatically generates appropriate questions and answers them. A sample set used for training the question-answer model is constructed based on the multiple sub-questions and answers, such as using a sub-question-answer pair as a sample data in the sample set.

[0082] The complex question and answer condition may be preset. For example, a target threshold corresponding to the complexity of the question is set, and the complex question and answer condition includes a complexity greater than the target threshold. The complexity of the question can be determined by the number of question characters, the number of fields involved, the amount of background information required, and the output requirements. For another example, at least one question and answer format belonging to a complex question is set, and the format of the complex question and answer condition conforms to any one of the at least one question and answer formats. A possible question and answer format is "the question information includes X question, the xx content in the X question means whether it is xxx, if it is xxx then how to xxxx, if not then xxxxx; the reply information includes: because of xxxx, so the xx in the question is xxx, further xxxxx, in addition to xxxx". The positions of the "X" and "x" symbols in the example question and answer format can be replaced with actual information, and the example is only used to represent the description format of the information. There may be other possible question and answer formats, or the actual question and answer format may also be different from the example, which is not limited here.

[0083] Question and answer pairs that meet complex question and answer conditions can be specified by staff, such as pre-specified when constructing questions; or can be judged by the construction device based on set complex question and answer conditions, such as the complex question and answer conditions including that the characters contained in the question information exceed the character count threshold, the content to be queried in the question information exceeds the target number, etc.

[0084] In one method, the construction device can perform diversity evolution processing on the generated question-answer pairs to avoid the situation where the question-answer pairs are obtained by filling in the template and the questions and answers may be rigid. For example, the question-answer pairs can be rewritten by the large model without changing the meaning of the original question-answer pairs, and rewritten into more complex, diverse and fluent question-answer pairs, and then the sample set used for training the question-answer model can be constructed based on the rewritten question-answer pairs.

[0085] In one method, the construction device can perform a thought chain expansion process on the generated question-answer pair. For example, for question information in the question-answer pair that cannot be directly answered, it is necessary to reason about the question information to obtain the answer information, guide the large model to generate a thinking process or a multi-step reasoning process for the question information, and determine the thought chain information based on which the answer information is obtained; then use the obtained thought chain information as sample data to construct a sample set used for training the question-answer model.

[0086] After obtaining the question-answer pairs required for training the question-answer model based on the annotation results, the construction device can also perform data evolution processing on these question-answer pairs through these methods to make the processed question-answer pairs of better quality and more conducive to model training.

[0087] In some embodiments, after obtaining the annotation results for the sample image, the construction device can also filter the annotation results to ensure that the annotation results with better quality are put into use for model training to avoid the model generating bad outputs. In the above step 208, the sample set is constructed based on the sample image, the global annotation results and the local annotation results, including: when the global annotation results and the local annotation results meet the screening conditions, the sample set is constructed based on the sample image, the global annotation results and the local annotation results; wherein the screening conditions include content compliance and / or the corresponding perplexity (PPL) is within the target range.

[0088] The construction device can remove harmful content, illegal content, and content suspected of being harmful in the global and local annotation results based on the screening conditions. For example, the construction device can use a large model to perform the screening process.

[0089] Perplexity is an important indicator to measure the performance of language models. It is mainly used to evaluate the model's predictive ability for text data and reflects the degree of uncertainty of the model when generating or predicting text. The lower the perplexity, the more accurate the model's prediction of the text; conversely, the higher the perplexity, the weaker the model's predictive ability. For images with low perplexity and the corresponding annotation results, it can be determined that the model itself can handle this type of data very well, so there is no need to further train the model based on these data to avoid wasting resources. For data with too high perplexity, there may be problems with the data itself and it is not suitable for model training. A sample set may not be constructed based on the data, and the data may not be used for model training. In some embodiments, for data with too high perplexity, it may also be that the model does not have the ability to process such data. It can be determined by human judgment whether the model needs to be trained with the corresponding ability to determine whether to use the data as sample data.

[0090] In some embodiments, the construction device may also screen the input sample images, and may filter out images with high resolution, images with abnormal aspect ratios, blurred images, and overexposed or dark images, without performing subsequent annotation processes on the images.

[0091] Figure 3 It is an overall implementation framework diagram of a sample set construction method provided in an embodiment of this specification. Figure 3 It can be cross-referenced with the introduction of the sample set construction method mentioned above. This solution mainly includes the stages of data intelligent labeling, instruction template construction and data evolution. The input of the data intelligent labeling stage is the original image, and the output is the overall image and detailed annotation text. It can also output the detection box or polygon corresponding to the target object. The input of the instruction template construction stage is the prompt template library, actual task scenarios and task requirements, industry background knowledge, etc., and the output is multiple backup prompt templates. The input of the data evolution stage is the basic question and answer pairs spliced ​​by templates and labeling results, and the output is the evolved sample set.

[0092] like Figure 3 As shown, the construction device can first be provided with the original image (such as the aforementioned sample image), the prompt template library, the scene / task requirement information corresponding to the model, and the industry background knowledge and other information. The construction device can perform overall pre-annotation on the original image to generate a global annotation result. It is also possible to detect and segment the target object on the original image, process the segmented detail image (that is, the aforementioned local view), and perform detail pre-annotation to obtain a local annotation result. Afterwards, the annotated results can be manually annotated and verified. During the manual annotation verification process, the verifier can use intelligent auxiliary annotation plug-ins such as interactive target segmentation plug-ins, auxiliary description generation plug-ins, and OCR plug-ins to quickly update the annotations.

[0093] The construction device can retrieve the backup prompt template that meets the requirements in the prompt template library based on the scenario / task requirement information corresponding to the model, and enhance the backup prompt template based on industry background knowledge to improve the fluency and richness of the template information. The construction device fills the annotation results obtained after verification into the target prompt template selected from the backup prompt template to construct a basic question and answer pair. Data evolution can be performed on the basic question and answer pair to obtain a higher quality question and answer pair, and then a sample set is constructed for training the model. The data evolution can include question and answer pair disassembly, diversity evolution, thinking chain expansion, and filtering of low-quality data. For these methods, please refer to the aforementioned related introduction, which will not be repeated here.

[0094] Figure 3 The image detection / segmentation, detail image processing, interactive target segmentation and OCR processes in the process can be implemented using the computer vision (CV) module. The overall pre-labeling, detail pre-labeling and auxiliary description generation processes can be implemented using the visual language model. The prompt template enhancement, question-answer pair disassembly, diversity evolution, thought chain expansion and filtering of low-quality data can be implemented using the large model.

[0095] In the embodiments of this specification, based on image detection and segmentation, multimodal large models and visual prompting technology, intelligent annotation of the sample image as a whole and detailed content can be performed to ensure the high-quality generation of the annotation results. The annotation process also supports a variety of interactive intelligent auxiliary annotation plug-ins, so that the annotation results can be verified using these plug-ins, which can improve the efficiency of manual verification and modification of annotations, and ensure the quality of the annotation results while greatly reducing the annotation cost, especially improving the comprehensiveness of the description of complex images and the quality of the detailed description.

[0096] In addition, in the embodiments of this specification, the question-answer pairs required for the training model are constructed by filling in the semi-structured prompt information template with information such as the annotation results and task requirements. It is possible to retrieve suitable industry instruction templates based on industry scenarios, task objectives, etc. and drive the large model to perform template enhancement and fill in the annotation results, thereby constructing a question-answer data set that is more suitable for improving the fine-tuning capabilities of the large model under the target scenario task. It is also possible to improve the quality and diversity of the data through various data evolution operators such as question-answer pair decomposition, diversity evolution, and thought chain expansion to ensure that the sample set is constructed with better results.

[0097] In summary, in the sample set construction method provided in the embodiment of this specification, on the basis of annotating the global view of the sample image, object recognition and corresponding local view capture are also performed on the sample image, and the image annotation model is instructed to locally annotate the image using visual prompt information for the global view and the local view, and the sample set is constructed based on the global annotation results and the local annotation results. In this way, a more comprehensive and complete annotation of the sample image can be ensured, and the accuracy and comprehensiveness of the annotation results can be improved. Accordingly, the sample set obtained based on the annotation results can achieve a better training effect on the model, ensuring a model with better performance.

[0098] The sample set constructed in the embodiments of this specification can be used to train a question-answering model. Figure 4 This is a flowchart of a method for training a question-answering model provided in one embodiment of the present specification. The training method can be applied to a model training device, which can be the same as the sample set construction device for executing the sample set construction method described above, or can be different, such as the training method can be executed by a cloud platform. In the case where the model training device is different from the sample set construction device, after the construction device has completed training the question-answering model by the model training device, the sample set construction device sends the constructed sample set to the model training device, and the model training device trains the question-answering model based on the sample set. Figure 4 As shown, the training method of the question-answering model includes the following steps 402 to 406.

[0099] Step 402: Obtain a sample set, wherein the sample set is constructed using the above-mentioned sample set construction method.

[0100] Step 402 can refer to the above Figure 2 and Figure 3 The sample set may include multiple sample data, each sample data may include a sample image and a corresponding annotation result, and the annotation result includes a global annotation result and a local annotation result.

[0101] Step 404: input the sample images in the sample set and the sample question information for the sample images into the initial question-answering model to predict reference answer information for the sample question information.

[0102] The initial question-answering model can analyze the sample image based on the input sample question information and predict the reference answer information of the sample question information. In some embodiments, the initial question-answering model is a question-answering model that has undergone a certain pre-training process. The question-answering model is fine-tuned for a specific industry based on the data in the aforementioned sample set to ensure that the question-answering model can be applied to accurate picture-text question-answering in the industry.

[0103] Step 406: Train the question-answering model based on the reference answer information, the global annotation results and the local annotation results of the sample images.

[0104] For the reference answer information of each sample image output by the initial question-answering model, it can be compared with the global annotation results and local annotation results of the sample image, and the parameters of the initial question-answering model can be adjusted based on the comparison results. Then the process of prediction, comparison and parameter adjustment is performed multiple times until the training stop condition is reached to obtain a trained question-answering model. Specifically, the loss value can be calculated based on the prediction result (that is, the reference answer information of the sample question information) and the annotation result of the sample image, and the model parameters of the initial question-answering model can be adjusted by backpropagation based on the loss value. Then return to execute the selection of sample images and sample question information, train the initial question-answering model until the training stop condition is reached, the training is completed, and a question-answering model that can be put into use is obtained. Among them, the training stop condition can be that the loss value is less than the loss value threshold, or it can also be that the number of iterations reaches the number threshold, or it can be combined with the loss value threshold and the number threshold to determine whether the training stop condition is reached.

[0105] The loss value threshold is the critical value of the loss value, and the value of the loss value threshold is selected according to the actual situation. When the loss value is greater than or equal to the loss value threshold, it means that there is still a certain deviation between the prediction result of the initial question and answer model and the actual result, and the model parameters of the initial question and answer model still need to be adjusted. At this time, it is determined that the training stop condition has not been met. When the loss value is less than the loss value threshold, it means that the prediction result of the initial question and answer model is close enough to the actual result, and the training can be stopped. At this time, it can be determined that the training stop condition is met. In addition, the number threshold is set according to the actual situation. When the number of iterations reaches the number threshold, it means that the initial question and answer model has been trained enough. At this time, the prediction result of the initial question and answer model is close to the actual result, and the training can be stopped.

[0106] In the embodiments of the present specification, the image annotation model used in the execution of the sample set construction method can directly adopt an existing image annotation model on the market, or can also be trained on a cloud platform.

[0107] In summary, on the basis of annotating the global view of the sample image, object recognition and corresponding local view capture are performed on the sample image, and the image annotation model is instructed to locally annotate the image using visual cue information for the global view and the local view, and a sample set is constructed based on the global annotation results and the local annotation results. In this way, a more comprehensive and complete annotation of the sample image can be ensured, and the accuracy and comprehensiveness of the annotation results can be improved. Accordingly, the sample set obtained based on the annotation results can achieve a better training effect on the model, ensuring a better performance question-answering model.

[0108] use Figure 4 The trained question-answering model can be applied to the question-answering process. Accordingly, a better question-answering processing effect can be obtained based on the question-answering model, and a more accurate and comprehensive response effect can be obtained for the question information. Figure 5 is a flowchart of a question-answering processing method provided in an embodiment of this specification, such as Figure 5 As shown, the method includes the following steps 502 and 504. Step 502: Obtain target question information for a target image. Step 504: Input the target image and the target question information into a question-answering model to obtain answer information of the target question information output by the question-answering model; wherein the question-answering model is trained based on the above-mentioned training method.

[0109] The question-and-answer processing method provided in the embodiments of this specification can be applied to a cloud platform. If the above-mentioned question-and-answer processing device is a cloud platform, the above-mentioned target question information can be sent by a terminal device. After obtaining the reply information of the target question information output by the question-and-answer model, the cloud platform can send the reply information to the terminal device for display by the terminal device. In some embodiments, the terminal device can also send question feedback information for the obtained reply information to the cloud platform. The cloud platform can adjust the question-and-answer model based on the question feedback information.

[0110] Figure 6 It is a flowchart of a request processing method based on a question-answer model provided in an embodiment of this specification, which is applied to a cloud platform and specifically includes the following steps 602 and 604.

[0111] Step 602: Receive a model request sent by a terminal device, wherein the model request includes at least one of a scene identifier of a target scene, scene input data of the target scene, and a target model specification parameter.

[0112] The cloud platform can be an online platform that provides model services, allowing developers, enterprises or individual users to submit tasks, obtain resources or implement specific functions through API calls, web interfaces or other interactive methods. In the embodiments of this specification, the cloud platform is a cloud service platform that can provide deep learning models, allowing external applications (such as e-commerce platforms, navigation applications, social applications, film and television production applications, and online education platforms, etc.) to access and use. The cloud platform ensures efficient and accurate response to requests from terminal devices by managing the training, updating, optimization, load balancing, scheduling and reasoning of the model. The terminal device is a terminal device that requests to obtain a question-and-answer model.

[0113] In actual implementation, a model request is a request sent by a terminal device to a cloud platform to obtain or use a specific model service. The model request may include a scene identifier of a target scene, scene input data of a target scene, and at least one of target model specification parameters. In the embodiments of this specification, the model request may be a request for a question-and-answer model.

[0114] Among them, the target scenario is the specific environment or situation where the question-answering model will be applied, which describes the purpose and expected effect of the model service. The scenario identifier of the target scenario is the identification information of the target scenario used for identification, which is used to assist in quickly locating the model related to the specific scenario, for example, "data annotation", "document question-answering", "image question-answering", "video question-answering", "video recommendation", "music recommendation", "news recommendation", "video navigation", "video monitoring", "video quality evaluation", "object positioning", "product positioning and search", etc. The scene input data of the target scenario can be the input data used to train the pre-trained question-answering model in the target scenario to adapt it to the target scenario, such as the scene input data in the image question-answering scenario can be a sample image with answer information. In addition, the model specification parameters are attribute specification parameters that define the model architecture, performance or configuration, such as model model, model type, model size, etc. The model specification parameters are used to determine the model that adapts to user needs.

[0115] For example, the user selects the scene identifier of the target scene as "Picture Question Answer" on the terminal device, selects the required model size as: 32GB, uploads the scene input data of the target scene, and the terminal device generates a model request and sends it to the cloud platform. The cloud platform receives the model request sent by the terminal device to the cloud platform, where the model request includes the scene identifier of the target scene, the scene input data of the target scene, and the target model specification parameters.

[0116] In the embodiments of the present specification, a user can independently select a scene identifier of a target scene, scene input data of a target scene, and at least one item of target model specification parameters on a terminal device to trigger a model request. The cloud platform can receive a model request for data annotation sent by the terminal device, and provide reference information for selecting a question-and-answer model based on the scene identifier of the target scene, scene input data of the target scene, and at least one item of target model specification parameters, which is applicable to a wider range of scenarios.

[0117] Step 604: Based on the model request, determine a corresponding question-answering model from at least one model, wherein the at least one model is trained based on the above-mentioned question-answering model training method, and the question-answering model is used to obtain answer information of the target question information.

[0118] In the embodiments of this specification, a model library may be provided, and the library may store at least one question-answering model adapted to different scenarios, and multiple question-answering models with different model specification parameters. For determining the corresponding question-answering model from at least one model based on a model request, one optional method is: based on a model request, searching for the corresponding question-answering model from at least one model included in the model library, which may store multiple question-answering models with different model specification parameters; another optional method is: based on a model request, training to obtain a question-answering model; another optional method is: based on a model request, constructing a question-answering model, which is not limited here.

[0119] In one implementation of step 604, determining the corresponding question-answering model from at least one model based on the model request may include: Based on at least one item of information in the model request, searching for a question-answering model that corresponds to the at least one item of information in at least one model in the model library; wherein the scene identifier of the target scene and the scene input data of the target scene correspond to the model adapted to the target scene, and the target model specification parameters correspond to the model whose model specification parameters are the same as the target model specification parameters; In a case where at least one item of information includes scene input data of a target scene, the found question-answering model is trained based on the scene input data of the target scene to obtain a trained question-answering model.

[0120] Regarding the implementation of step 604, the cloud platform can determine the required question-answering model by searching for a model from a model library based on the scene identifier of the target scene and the target model specification parameters. The required question-answering model is determined by searching for a model from a model library based on the scene input data of the target scene and continuing to train the found model.

[0121] In the case where the model request includes only one item of information, if the model request includes the scene identifier of the target scene, step 604 may include: based on the scene identifier of the target scene, searching the model library for a question-and-answer model suitable for the target scene, wherein the model library stores at least one question-and-answer model suitable for different scenes. If the model request includes the scene input data of the target scene, step 604 may include: determining an initial question-and-answer model suitable for the target scene from at least one model; training the initial question-and-answer model based on the scene input data of the target scene to obtain a trained question-and-answer model. If the model request includes model specification parameters, step 604 may include: based on the model specification parameters, searching the model library for a corresponding question-and-answer model, wherein the model library stores multiple question-and-answer models with different model specification parameters.

[0122] When the model request includes two or three items of information including the scene identification of the target scene, the scene input data of the target scene, and the target model specification parameters, the cloud platform can search for a model that satisfies the two or three items of information in the model library to determine the required model. If the model request includes the scene identification of the target scene and the target model specification parameters, the cloud platform can search for a model corresponding to the target model specification parameters that are suitable for the target scene in the model library. If the model request includes the scene identification of the target scene and the scene input data of the target scene, the cloud platform can first search for a model that is suitable for the target scene in the model library, and then train the model based on the scene input data of the target scene to obtain the required model. If the model request includes the target model specification parameters and the scene input data of the target scene, the cloud platform can first search for a model of the target model specification parameters in the model library, and then train the model based on the scene input data of the target scene to obtain the required model.

[0123] For the case where the model request includes the scene identification of the target scene, the scene input data of the target scene, and the target model specification parameters, the cloud platform can first search for at least one pre-trained question and answer model from the model library based on the scene identification of the target scene, and then based on the target model specification parameters, screen the question and answer model with the target model specification parameters from at least one question and answer model, and then train the question and answer model with the target model specification parameters based on the scene input data of the target scene to obtain a question and answer model suitable for user needs.

[0124] The question-answering model obtained by the cloud platform is based on Figure 4 The training method of the above-mentioned question-answering model is obtained by training. For the specific method, please refer to the content of the above-mentioned model training, which will not be repeated here.

[0125] In the embodiments of this specification, based on the scenario requirements, the question and answer models suitable for the scenario are accurately found through scenario identification, and then these models are used to generate answer information corresponding to the question information. This can ensure that the answer information corresponding to the obtained question information is more accurate and fits the scenario, thereby improving the user experience and question and answer quality.

[0126] Optionally, for different scenarios, the answer information corresponding to the required question information may be different. As an example, the cloud platform can provide question-answering models for multiple scenarios, and can provide corresponding question-answering models based on the model request for data annotation sent by the terminal device, so as to use the question-answering model to determine the accurate and effective answer information corresponding to the target question information.

[0127] In the embodiment of this specification, after step 604, the cloud platform may also deploy a question-answering model, and based on the question-answering model, construct a calling interface (such as the calling interface may be a question-answering interface), so that the terminal device may call the question-answering model through the calling interface to obtain the answer information of the target question information. For example, the cloud platform may deploy the question-answering model in a virtual machine, a container orchestration platform, an edge node, or a dedicated processor. When the terminal device calls, the device where the model is deployed is triggered to run the corresponding program or code to achieve the answer to the target question information.

[0128] Corresponding to the above method embodiment, this specification also provides a task platform embodiment, Figure 7 Schematic diagram of a task platform provided in one embodiment of this specification. Figure 7 As shown, the task platform 700 includes: a request interface 702 and a response unit 704; A request interface 702, used to receive a model request sent by a terminal device, wherein the model request includes at least one of a scene identifier of a target scene, scene input data of the target scene, and a model specification parameter; The response unit 704 is used to determine the corresponding question-answering model from at least one model based on the model request, wherein the at least one model is trained based on the training method of the above-mentioned question-answering model.

[0129] Optionally, the model library stores at least one question-answering model adapted to different scenarios, and multiple question-answering models with different model specification parameters; the response unit 704 is used to: based on at least one item of information in the model request, search the model library for a question-answering model that corresponds to at least one item of information; wherein the scene identifier of the target scene and the scene input data of the target scene correspond to the model adapted to the target scene, and the target model specification parameters correspond to the model whose model specification parameters are the same as the target model specification parameters; when at least one item of information includes the scene input data of the target scene, the found question-answering model is trained based on the scene input data of the target scene to obtain a trained question-answering model.

[0130] Optionally, the task platform also includes a calling interface, which is constructed based on the question-answering model; the calling interface is used for the terminal device to call and obtain answer information of the target question information.

[0131] The above is a schematic scheme of a task platform of this embodiment. It should be noted that the technical scheme of the task platform and the technical scheme of the above-mentioned sample set construction method belong to the same concept, and the details not described in detail in the technical scheme of the task platform can be found in the description of the technical scheme of the above-mentioned question and answer processing method.

[0132] In some implementations, there may not be an inference model (i.e., a question-answering model) that can execute the question-answering processing method. In this case, some existing models need to be trained to obtain the inference model. The embodiments of this specification also provide a model training platform, on which model training can be performed to obtain the required inference model.

[0133] The model training platform includes a request interface unit, a model training unit and a response unit. The model training platform can refer to the previous introduction to the task platform. The request interface unit is used to receive a model training request, wherein the model training request includes model attribute information of the machine learning model to be trained; the model training unit is used to determine the machine learning model to be trained according to the model attribute information, and perform model training on the machine learning model to be trained to obtain a target model, wherein the target model is obtained by training based on the above training method. The response unit is used to output the target model.

[0134] Optionally, the model training platform further includes a data receiving unit for receiving training data input by a user and sending the training data to the model training unit. The model training unit is further used to perform model training on a target model according to the training data to obtain a trained target model. Optionally, the model training platform further includes a model library, wherein the model library stores a plurality of machine learning models; the model training unit is further used to determine a machine learning model to be trained from the model library according to the model attribute information.

[0135] The example of this specification also provides a model training method, which is applied to a model training platform, including: receiving a model training request, wherein the model training request includes model attribute information of a machine learning model to be trained; determining the machine learning model to be trained based on the model attribute information, and performing model training on the machine learning model to be trained to obtain a target model, wherein the target model is obtained by training using the above-mentioned model training method; and outputting the target model.

[0136] Corresponding to the above-mentioned sample set construction method embodiment, this specification also provides a sample set construction device embodiment, Figure 8 Schematic diagram of a sample set construction device provided in an embodiment of this specification. Figure 8 As shown, the sample set construction device includes: A first acquisition module 802 is used to input the sample image into the image annotation model to obtain a global annotation result of the sample image; The second acquisition module 804 is used to identify the target object in the sample image, and based on the obtained recognition result, mark the target object in the sample image to obtain a global view, and intercept the target object to obtain a local view; A third acquisition module 806 is used to input the global view, the local view and the visual prompt information into the image annotation model to obtain the local annotation result of the sample image, wherein the visual prompt information is used to prompt the image annotation model to perform the annotation task on the global view based on the local view; The first construction module 808 is used to construct a sample set based on the sample image, the global annotation result and the local annotation result.

[0137] Optionally, the sample set construction device further includes: The second construction module is used to construct preset prompt information based on task requirement information before inputting the sample image into the image annotation model to obtain the global annotation result of the sample image; wherein the task requirement information indicates the target task to be performed by the model to be trained; the preset prompt information is used to prompt the image annotation model to extract information of the input image for global annotation according to the target content type corresponding to the target task; The first acquisition module 802 is used to: input the sample image and preset prompt information into the image annotation model, so that the image annotation model extracts the information of the sample image according to the target content type corresponding to the target task, and obtains the global annotation result of the sample image, wherein the global annotation result includes the information of the target content type.

[0138] Optionally, the sample set construction device also includes: a third construction module, which is used to construct visual prompt information based on task requirement information before inputting the global view, local view and visual prompt information into the image annotation model, wherein the task requirement information indicates the target task that the model to be trained is used to perform, the visual prompt information is used to prompt the image annotation model to perform the annotation task, extract information from the input local view according to the target content type corresponding to the target task, and determine the association information between the input image local view and the global view.

[0139] Optionally, the first construction module 808 is used to: send the sample image, global annotation results and local annotation results to the front end, and receive annotation verification information for the sample image sent by the front end; when the annotation verification information indicates that the local annotation result has not passed the verification and carries the regional information of the area where the target object is located, intercept the auxiliary area indicated by the regional information in the sample image to obtain an updated local view; input the global view, the updated local view and the visual prompt information into the image annotation model to obtain the updated local annotation result of the sample image; wherein the visual prompt information is used to prompt the image annotation model to perform the annotation task on the global view based on the updated local view; and construct a sample set based on the sample image, the global annotation result and the newly added local annotation result.

[0140] Optionally, the first building module 808 is used to: Send the sample image, global annotation results and local annotation results to the front end, and receive the annotation verification information for the sample image sent by the front end; When the annotation verification information indicates that the global annotation result has not passed the verification and carries the first prompt information, the sample image and the first prompt information are input into the image annotation model to obtain an updated global annotation result of the sample image; wherein the first prompt information is used to prompt the image annotation model to extract information of the input image for global annotation according to the target content type; When the annotation verification information indicates that the local annotation result has not passed the verification and carries the second prompt information, the global view, the local view and the second prompt information are input into the image annotation model to obtain an updated local annotation result of the sample image; wherein the second prompt information is used to prompt the image annotation model to perform the annotation task on the global view based on the local view, and the second prompt information is different from the visual prompt information; Based on the sample images, the global annotation results are updated and the local annotation results are updated to construct a sample set.

[0141] Optionally, the first building module 808 is used to: Fill the global annotation results and the local annotation results into the target prompt information template; wherein the target prompt information template includes a content part, an instruction part and an output part, the content part indicates the target task to be performed by the question-answering model and the information of the scene it targets, the instruction part indicates the task instruction to be performed by the question-answering model, and the global annotation results and the local annotation results are filled into the output part; Based on the information in the filled target prompt information template, a question-answer pair is generated, and a sample set used for training the question-answer model is constructed based on the question-answer pair; wherein the question information in the question-answer pair includes information of the content part and the instruction part, and the answer information in the question-answer pair includes information of the output part.

[0142] Optionally, the sample set construction device further includes: A query module, used for searching for a spare prompt information template whose similarity with the task requirement information is higher than a target threshold among multiple prompt information templates before filling the global annotation result and the local annotation result into the target prompt information template; wherein the task requirement information indicates the target task to be performed by the question-answering model; The determination module is used to use any spare prompt information template as a target prompt information template when at least one spare prompt information template is found.

[0143] Optionally, the sample set construction device also includes: a rewriting module, which is used to fill the task requirement information and the information of the scene it targets into the content part of the target prompt information template before filling the global annotation results and the local annotation results into the target prompt information template, and use the text adjustment model to adjust the content in the target prompt information template.

[0144] Optionally, the first building module 808 is configured to perform at least one of the following steps: When the question-answer pair meets the complex question-answer condition, the question-answer pair is decomposed to obtain multiple sub-question-answer pairs; a sample set used for training the question-answer model is constructed based on the multiple sub-questions and answers; wherein the complex question-answer condition includes that the complexity is greater than the target threshold, and / or the format meets the set question-answer format; Using the text adjustment model, the question-answer pair is rewritten, and the semantics of the rewritten question-answer pair remains unchanged; based on the rewritten question-answer pair, a sample set used for training the question-answer model is constructed; When the answer information in a question-answer pair needs to be inferred based on the question information, thought chain information based on which the answer information is derived is generated based on the question information; based on the thought chain information and the answer information, a sample set used to train the question-answering model is constructed.

[0145] Optionally, the first construction module 808 is used to: construct a sample set based on the sample image, the global annotation results and the local annotation results when the global annotation results and the local annotation results meet the screening conditions; wherein the screening conditions include content compliance and / or the corresponding perplexity is within a target range.

[0146] The sample set construction device of the embodiment of this specification can ensure more comprehensive and complete annotation of sample images, improve the accuracy and comprehensiveness of the annotation results. Accordingly, the sample set obtained based on the annotation results can achieve better training effects on the model, ensuring a model with better performance.

[0147] Fig. 9 is a schematic diagram of a training device for a question-answering model provided in an embodiment of this specification. The training device can be used with Figure 4 The training methods shown are cross-referenced. Fig. 9 As shown, the training device comprises: A sample set acquisition module 902 is used to acquire a sample set, wherein the sample set is constructed using the above-mentioned sample set method; Prediction module 904, used to input sample images in the sample set and sample question information for the sample images into the initial question-answering model, and predict reference answer information for the sample question information; The training module 906 is used to train the question-answering model based on the reference answer information, the global annotation results and the local annotation results of the sample images.

[0148] Fig.10 is a schematic diagram of a question-answering processing device provided in an embodiment of the present specification. The question-answering processing device can be used with Figure 5 The question-answer processing methods shown in the figure refer to each other. Fig.10 As shown, the question-answer processing device includes: The question acquisition module 1002 is used to acquire target question information for the target image; The answer module 1004 is used to input the target image and the target question information into the question-answering model, and obtain the answer information of the target question information output by the question-answering model; wherein the question-answering model is based on Figure 4 The training method shown is used for training.

[0149] Fig.11 It is a structural block diagram of a computing device provided in an embodiment of this specification. The computing device 1100 includes: a memory 1110 and a processor 1120; the memory 1110 is used to store computer programs / instructions, and the processor 1120 is used to execute the computer programs / instructions, and the steps of the above method are implemented when the computer programs / instructions are executed by the processor 1120. In one or more embodiments of this specification, the computing device can be understood as an integrated intelligent terminal, including but not limited to a server, a desktop computer, a PC (Personal Computer), a model all-in-one machine, a mobile phone, a tablet computer or other portable intelligent terminal, etc., and the computing device can be pre-installed with the model described in the above embodiments of this application.

[0150] Specifically, the computing device can pre-set multiple types of models, including but not limited to models in the fields of natural language processing, visual processing, speech processing, code processing, multimodal task processing, etc., so as to provide a variety of model choices. In different product forms, the computing device can support one or more model usage methods, including but not limited to model training, model calling, model fine-tuning, model deployment, model reasoning and application, etc. In some product forms, the computing device also supports model management, including but not limited to multi-type model management (supporting the management of multiple types of models such as discriminants and generative models), model version control (supporting the control of different model versions), model evaluation (based on model evaluation tools, evaluating the performance and effect of the model), etc. In other product forms, the computing device can also create applications based on the model, provide API (Application Programming Interface, application programming interface) calling capabilities, and can call the model to the created application through the API interface, while providing application management tools to achieve management and monitoring of the application.

[0151] Furthermore, the computing device can also include data management (supporting the creation and management of model tuning data sets), training center (providing rich training resources to help users learn and master artificial intelligence (AI) technology), and basic management and control capabilities (providing enterprise-level basic management and control capabilities to ensure the security and efficient operation of the system). Through the above functions, a comprehensive, integrated AI development, training, deployment and application device is provided.

[0152] Fig.12 1 is a block diagram of an electronic device provided in an embodiment of the present specification. The electronic device 1200 includes a memory 1210 and a processor 1220, and the memory 1210 and the processor 1220 are connected via a bus; the memory 1210 is used to store computer programs / instructions, and the processor 1220 is used to execute computer programs / instructions, and the steps of the above method are implemented when the computer program / instructions are executed by the processor 1220. Specifically, the components of the electronic device 1200 include but are not limited to the memory 1210 and the processor 1220. The processor 1220 and the memory 1210 can be connected via a bus 1230.

[0153] The electronic device 1200 may also include an access device 1240, which enables the electronic device 1200 to communicate with a database 1250 storing data via one or more networks 1260. Examples of these networks include a public switched telephone network (PSTN), a local area network (LAN), a wide area network (WAN), a personal area network (PAN), or a combination of communication networks such as the Internet. The access device may include one or more of any type of network interface (e.g., a network interface card (NIC)) of wired or wireless, such as an IEEE 802.11 wireless local area network (WLAN) wireless interface, a world-wide interoperability for microwave access (Wi-MAX) interface, an Ethernet interface, a universal serial bus (USB) interface, a cellular network interface, a Bluetooth interface, and a near field communication (NFC).

[0154] In one embodiment of the present specification, the above components of the electronic device 1200 and Fig.12Other components not shown in the figure may also be connected to each other, for example, via a bus. It should be understood that Fig.12 The electronic device structure block diagram shown is only for the purpose of illustration, and is not intended to limit the scope of this specification. Those skilled in the art can add or replace other components as needed. The electronic device 1200 can be any type of stationary or mobile electronic device, including a mobile computer or mobile electronic device (e.g., a tablet computer, a personal digital assistant, a laptop computer, a notebook computer, a netbook, etc.), a mobile phone (e.g., a smart phone), a wearable electronic device (e.g., a smart watch, smart glasses, etc.) or other types of mobile devices, or a stationary electronic device such as a desktop computer or a personal computer (PC, Personal Computer). The electronic device 1200 can also be a mobile or stationary server.

[0155] An embodiment of the present specification further provides a computer-readable storage medium storing a computer program / instruction, which implements the steps of the above method when executed by a processor.

[0156] An embodiment of the present specification also provides a computer program product, including a computer program / instruction, which implements the steps of the above method when executed by a processor.

[0157] Computer programs / instructions include computer program codes, which may be in source code form, object code form, executable files or some intermediate forms, etc. Computer readable media may include: any entity or device capable of carrying computer program codes, recording media, USB flash drives, mobile hard disks, magnetic disks, optical disks, computer memories, read-only memories (ROM), random access memories (RAM), electric carrier signals, telecommunication signals, and software distribution media, etc. It should be noted that the contents of computer readable media may be appropriately increased or decreased according to the requirements of patent practice. For example, in some regions, according to patent practice, computer readable media do not include electric carrier signals and telecommunication signals.

[0158] The technical solutions of the above-mentioned computing device, computer-readable storage medium and computer program product belong to the same concept as the above-mentioned sample set construction method. For details not described in detail in the technical solution of the computer program product, please refer to the description of the technical solution of the above-mentioned method. The above describes a specific embodiment of this specification. Other embodiments are within the scope of the attached claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the accompanying drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0159] It should be noted that, for the aforementioned method embodiments, for the sake of simplicity of description, they are all expressed as a series of action combinations, but those skilled in the art should be aware that the embodiments of this specification are not limited by the described order of actions, because according to the embodiments of this specification, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the embodiments of this specification. In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, please refer to the relevant description of other embodiments.

[0160] The preferred embodiments of this specification disclosed above are only used to help explain this specification. The optional embodiments do not describe all the details in detail, nor do they limit the invention to only the specific implementation methods described. Obviously, many modifications and changes can be made according to the content of the embodiments of this specification. This specification selects and specifically describes these embodiments in order to better explain the principles and practical applications of the embodiments of this specification, so that technicians in the relevant technical field can well understand and use this specification. This specification is only limited by the claims and their full scope and equivalents.

Claims

1. A sample set construction method, comprising: Inputting the sample image into the image annotation model to obtain a global annotation result of the sample image; Identify the target object in the sample image, and based on the obtained recognition result, mark the target object in the sample image to obtain a global view, and intercept the target object to obtain a local view; Inputting the global view, the local view and the visual cue information into the image annotation model to obtain a local annotation result of the sample image, wherein the visual cue information is used to prompt the image annotation model to perform an annotation task on the global view based on the local view; A sample set is constructed based on the sample image, the global annotation result and the local annotation result.

2. The method according to claim 1, before inputting the sample image into the image annotation model to obtain the global annotation result of the sample image, further comprising: Based on the task requirement information, construct preset prompt information; wherein the task requirement information indicates the target task that the model to be trained is used to perform; the preset prompt information is used to prompt the image annotation model to extract information of the input image for global annotation according to the target content type corresponding to the target task; The step of inputting the sample image into the image annotation model to obtain a global annotation result of the sample image includes: The sample image and the preset prompt information are input into the image annotation model, so that the image annotation model extracts the information of the sample image according to the target content type corresponding to the target task, and obtains the global annotation result of the sample image; wherein the global annotation result includes the information of the target content type.

3. The method according to claim 1, before inputting the global view, the local view and the visual cue information into the image annotation model, further comprising: Based on the task requirement information, the visual prompt information is constructed; wherein the task requirement information indicates the target task that the model to be trained is used to perform; the visual prompt information is used to prompt the image annotation model to perform the annotation task, extract information from the input local view according to the target content type corresponding to the target task, and determine the association information between the input image local view and the global view.

4. The method according to any one of claims 1 to 3, wherein constructing a sample set based on the sample image, the global annotation result and the local annotation result comprises: Sending the sample image, the global annotation result, and the local annotation result to a front end, and receiving annotation verification information for the sample image sent by the front end; When the annotation verification information indicates that the local annotation result has not passed the verification and carries the region information of the region where the target object is located, intercepting the auxiliary region indicated by the region information in the sample image to obtain an updated local view; Inputting the global view, the updated local view and the visual prompt information into the image annotation model to obtain an updated local annotation result of the sample image; wherein the visual prompt information is used to prompt the image annotation model to perform an annotation task on the global view based on the updated local view; The sample set is constructed based on the sample image, the global annotation result and the newly added local annotation result.

5. The method according to any one of claims 1 to 3, wherein constructing a sample set based on the sample image, the global annotation result and the local annotation result comprises: Sending the sample image, the global annotation result, and the local annotation result to a front end, and receiving annotation verification information for the sample image sent by the front end; When the annotation verification information indicates that the global annotation result has not passed the verification and carries the first prompt information, the sample image and the first prompt information are input into the image annotation model to obtain an updated global annotation result of the sample image; wherein the first prompt information is used to prompt the image annotation model to extract information of the input image for global annotation according to the target content type; When the annotation verification information indicates that the local annotation result has not passed the verification and carries the second prompt information, the global view, the local view and the second prompt information are input into the image annotation model to obtain an updated local annotation result of the sample image; wherein the second prompt information is used to prompt the image annotation model to perform the annotation task on the global view based on the local view, and the second prompt information is different from the visual prompt information; The sample set is constructed based on the sample image, the updated global annotation result and the updated local annotation result.

6. The method according to any one of claims 1 to 3, wherein constructing a sample set based on the sample image, the global annotation result and the local annotation result comprises: Fill the global annotation result and the local annotation result into a target prompt information template; wherein the target prompt information template includes a content part, an instruction part and an output part, the content part indicates the target task to be performed by the question-answering model and the information of the scene it targets, the instruction part indicates the task instruction to be performed by the question-answering model, and the global annotation result and the local annotation result are filled into the output part; Based on the information in the filled target prompt information template, a question-answer pair is generated, and a sample set used to train the question-answer model is constructed based on the question-answer pair; wherein the question information in the question-answer pair includes information of the content part and the instruction part, and the answer information in the question-answer pair includes information of the output part.

7. The method according to claim 6, before filling the global labeling result and the local labeling result into the target prompt information template, further comprising: Searching for a spare prompt information template whose similarity with the task requirement information is higher than a target threshold among multiple prompt information templates; wherein the task requirement information indicates a target task for the question-answering model to perform; In the case where at least one of the backup prompt information templates is found, any of the backup prompt information templates is used as the target prompt information template.

8. The method according to claim 6, before filling the global labeling result and the local labeling result into the target prompt information template, further comprising: The task requirement information and the information of the scene it targets are filled into the content part of the target prompt information template, and the content of the target prompt information template is adjusted using a text adjustment model.

9. The method according to claim 6, wherein constructing a sample set for training the question-answer model based on the question-answer pairs comprises at least one of the following steps: When the question-answer pair satisfies the complex question-answer condition, the question-answer pair is decomposed to obtain multiple sub-question-answer pairs; a sample set used for training the question-answer model is constructed based on the multiple sub-questions and answers; wherein, The complex question and answer conditions include that the complexity is greater than a target threshold, and / or the format satisfies a set question and answer format; Rewriting the question-answer pair using a text adjustment model, and making the semantics of the rewritten question-answer pair unchanged; Constructing a sample set used for training the question-answer model based on the rewritten question-answer pairs; In the case where the answer information in the question-answer pair needs to be inferred based on the question information, thought chain information based on which the answer information is derived is generated for the question information; based on the thought chain information and the answer information, a sample set used to train the question-answer model is constructed.

10. The method according to any one of claims 1 to 3, wherein constructing a sample set based on the sample image, the global annotation result and the local annotation result comprises: When the global annotation result and the local annotation result meet the screening conditions, the sample set is constructed based on the sample image, the global annotation result and the local annotation result; wherein the screening conditions include content compliance and / or the corresponding perplexity being within a target range.

11. A method for training a question-answering model, comprising: Acquire a sample set, wherein the sample set is constructed using the method according to any one of claims 1 to 10; Inputting sample images in the sample set and sample question information for the sample images into an initial question-answering model to predict reference answer information for the sample question information; The question-answering model is trained based on the reference answer information, the global annotation result and the local annotation result of the sample image.

12. A question-answering processing method, comprising: Obtain target question information for a target image; The target image and the target question information are input into a question-answering model to obtain answer information of the target question information output by the question-answering model; wherein the question-answering model is trained based on the training method described in claim 11.

13. A request processing method based on a question-answering model, applied to a task platform, comprising: Receiving a model request sent by a terminal device, wherein the model request includes at least one of a scene identifier of a target scene, scene input data of the target scene, and a target model specification parameter; Based on the model request, a corresponding question-answering model is determined from at least one model; wherein the at least one model is trained based on the training method according to claim 11, and the question-answering model is used to obtain answer information of the target question information.

14. The method according to claim 13, wherein the model library stores at least one question-answering model adapted to different scenarios and a plurality of question-answering models with different model specification parameters; and determining the corresponding question-answering model from at least one model based on the model request comprises: Based on the at least one item of information in the model request, searching for a question-answering model corresponding to the at least one item of information in at least one model in the model library; wherein the scene identifier of the target scene and the scene input data of the target scene correspond to a model adapted to the target scene, and the target model specification parameters correspond to a model having the same model specification parameters as the target model specification parameters; In a case where the at least one item of information includes scene input data of the target scene, the found question-answering model is trained based on the scene input data of the target scene to obtain a trained question-answering model.

15. The method according to claim 13 or 14, after determining a corresponding question-answering model from at least one model based on the model request, further comprising: The question-answering model is deployed, and based on the question-answering model, a calling interface is constructed so that the terminal device calls the question-answering model through the calling interface to obtain answer information corresponding to the target question information.

16. A task platform, comprising a request interface and a response unit; The request interface is used to receive a model request sent by a terminal device, wherein: The model request includes at least one of a scene identifier of a target scene, scene input data of the target scene, and a model specification parameter; The response unit is used to determine a corresponding question-answering model from at least one model based on the model request, wherein the at least one model is trained based on the training method described in claim 11.

17. The task platform according to claim 16 further includes a calling interface, which is constructed based on the question and answer model; the calling interface is used for the terminal device to call and obtain reply information of the target question information.

18. A computing device comprising: Memory and processor; The memory is used to store computer programs / instructions, and the processor is used to execute the computer programs / instructions. When the computer programs / instructions are executed by the processor, the steps of the method described in any one of claims 1 to 15 are implemented.

19. An electronic device comprising: A memory and a processor, wherein the memory and the processor are connected via a bus; The memory is used to store computer programs / instructions, and the processor is used to execute the computer programs / instructions. When the computer programs / instructions are executed by the processor, the steps of the method described in any one of claims 1 to 15 are implemented.

20. A computer-readable storage medium storing a computer program / instruction, wherein the computer program / instruction, when executed by a processor, implements the steps of the method according to any one of claims 1 to 15.

21. A computer program product, comprising a computer program / instructions, which, when executed by a processor, implement the steps of the method according to any one of claims 1 to 15.

Images