Multi-modal generic perception model training method and apparatus, labeling method and apparatus, and electronic device

Abstract

The present disclosure relates to the technical field of data processing, and disclosed are a multi-modal generic perception model training method and apparatus, a labeling method and apparatus, and an electronic device. The method comprises: determining a plurality of sample images, labeling each sample image, and determining a plurality of initial region-level image-text data labeling results corresponding to each sample image; correcting the plurality of initial region-level image-text data labeling results corresponding to each sample image, and determining a plurality of corrected labeling results corresponding to each sample image; and on the basis of the plurality of corrected labeling results corresponding to each sample image, training a multi-modal generic perception model to obtain a trained multi-modal generic perception model, wherein the trained multi-modal generic perception model is used for labeling a target image and determining a plurality of region-level image-text data labeling results corresponding to the target image.

Description

Multi-modal general perception model training, labeling method and device, and electronic device

[0001] Cross-reference to Related Applications

[0002] The present application is based on the Chinese patent application No. CN202410802858.3, filed on June 20, 2024, and claims the priority of the Chinese patent application, the whole content of which is incorporated herein by reference. TECHNICAL FIELD

[0003] The present disclosure relates to the technical field of data processing, and in particular to a multi-modal general perception model training, labeling method and device, and electronic device. BACKGROUND

[0004] Region-level image-text data labeling is different from the commonly used image-text data labeling for the whole sample image, and can separately label a target region and / or target object in the sample image, and has high directivity. In the prior art, the region-level image-text data labeling of the sample image is usually realized by using the artificial labeling method to determine the corresponding region-level image-text labeling result. However, due to the high efficiency and cost of artificial labeling, it is difficult to quickly and effectively obtain a large number of region-level image-text data labeling results. SUMMARY

[0005] Therefore, the present disclosure provides a multi-modal general perception model training, labeling method and device, and electronic device.

[0006] According to an aspect of the present disclosure, a multi-modal general perception model training method is provided, comprising: determining a plurality of sample images, and respectively labeling each sample image to determine a plurality of initial region-level image-text data labeling results corresponding to each sample image; correcting the plurality of initial region-level image-text data labeling results corresponding to each sample image to determine a plurality of corrected labeling results corresponding to each sample image; and training a multi-modal general perception model based on the plurality of corrected labeling results corresponding to each sample image to obtain a trained multi-modal general perception model, wherein the trained multi-modal general perception model is used to label a target image to determine a plurality of region-level image-text data labeling results corresponding to the target image, and any one of the region-level image-text data labeling results corresponding to the target image represents a separate labeling result of any one of the target objects in the target image.

[0007] In some possible implementation manners, the labeling each sample image respectively to determine a plurality of initial region-level image-text data labeling results corresponding to each sample image comprises: for any one sample image, determining position information corresponding to each target object in the sample image according to the position recognition model respectively; for any one target object in the sample image, determining candidate semantic information corresponding to the target object according to the feature extraction model; for any one target object in the sample image, pairing the position information and the candidate semantic information corresponding to the target object according to the image-text alignment model to obtain pairing information corresponding to the target object; for any one target object in the sample image, determining question-answer pair information and description information corresponding to the target object according to the pairing information corresponding to the target object, wherein the question-answer pair information corresponding to the target object represents a question and an answer related to the target object, and the description information corresponding to the target object is used to describe features and states of the target object; and determining the plurality of initial region-level image-text data labeling results corresponding to the sample image according to the pairing information, the question-answer pair information and the description information corresponding to each target object in the sample image.

[0008] In some possible implementation manners, the correcting the plurality of initial region-level image-text data labeling results corresponding to each sample image to determine a plurality of corrected labeling results corresponding to each sample image comprises: dividing all initial region-level image-text data labeling results corresponding to the plurality of sample images into a first sample group and a second sample group, wherein a number of initial region-level image-text data labeling results included in the first sample group is greater than a number of initial region-level image-text data labeling results included in the second sample group; and correcting each initial region-level image-text data labeling result included in the second sample group respectively to determine a corrected second sample group.

[0009] In some possible implementation manners, the correcting each initial region-level image-text data labeling result included in the second sample group respectively to determine the corrected second sample group comprises: deleting incorrect candidate semantic information in each initial region-level image-text data labeling result included in the second sample group to obtain a second sample group after semantic verification; and adjusting incorrect question-answer pair information in each initial region-level image-text data labeling result included in the second sample group after semantic verification to determine the corrected second sample group.

[0010] In some possible implementation manners, the training of the multi-modal general perception model based on the plurality of corrected annotation results corresponding to each sample image comprises: pre-training the multi-modal general perception model according to the first sample group to obtain a pre-trained multi-modal general perception model; and fine-tuning the pre-trained multi-modal general perception model according to the corrected second sample group to determine the trained multi-modal general perception model.

[0011] According to another aspect of the present disclosure, a labeling method is provided, comprising: determining a target image, wherein the target image comprises at least one target object; and automatically labeling the target image according to a multi-modal general perception model to determine a plurality of region-level image-text data annotation results corresponding to the target image, wherein the multi-modal general perception model is trained by the method described above, and any one of the region-level image-text data annotation results corresponding to the target image represents an individual annotation result of any one of the target objects in the target image.

[0012] According to another aspect of the present disclosure, a multi-modal general perception model training apparatus is provided, comprising: an initial labeling module configured to determine a plurality of sample images, and label each sample image to determine a plurality of initial region-level image-text data annotation results corresponding to each sample image; a correction module configured to correct the plurality of initial region-level image-text data annotation results corresponding to each sample image to determine a plurality of corrected annotation results corresponding to each sample image; and a model training module configured to train a multi-modal general perception model based on the plurality of corrected annotation results corresponding to each sample image to obtain a trained multi-modal general perception model, wherein the trained multi-modal general perception model is configured to label a target image to determine a plurality of region-level image-text data annotation results corresponding to the target image, and any one of the region-level image-text data annotation results corresponding to the target image represents an individual annotation result of any one of the target objects in the target image.

[0013] According to another aspect of the present disclosure, a labeling apparatus is provided, comprising: an image acquisition module configured to determine a target image, wherein the target image comprises at least one target object; and a labeling module configured to automatically label the target image according to a multi-modal general perception model to determine a plurality of region-level image-text data annotation results corresponding to the target image, wherein the multi-modal general perception model is trained by the method described above, and any one of the region-level image-text data annotation results corresponding to the target image represents an individual annotation result of any one of the target objects in the target image.

[0014] According to another aspect of the present disclosure, an electronic device is provided, comprising: a processor; a memory for storing processor-executable instructions; wherein the processor is configured to implement the above method when executing the instructions stored in the memory.

[0015] According to another aspect of the present disclosure, a non-volatile computer-readable storage medium having stored thereon computer program instructions is provided, wherein the computer program instructions, when executed by a processor, implement the above method.

[0016] In the embodiments of the present disclosure, by determining a plurality of sample images and respectively labeling each sample image, a plurality of initial region-level image-text data labeling results corresponding to each sample image can be determined; by correcting the plurality of initial region-level image-text data labeling results corresponding to each sample image, a plurality of corrected labeling results corresponding to each sample image can be determined, thereby improving the accuracy of the corrected labeling results, so that any one of the corrected labeling results can accurately describe the features and states of the target object corresponding to the corrected labeling result; based on the plurality of corrected labeling results corresponding to each sample image, the multi-modal general perception model is trained, and a trained multi-modal general perception model can be obtained, and the trained multi-modal general perception model can be used to label a target image and determine a plurality of region-level image-text data labeling results corresponding to the target image, any one of the region-level image-text data labeling results corresponding to the target image representing an individual labeling result of any one target object in the target image, thereby simplifying the process of region-level image-text labeling, reducing the participation of manual operation in the labeling process, reducing the workload and cost of region-level image-text labeling, improving the working efficiency of region-level image-text labeling, and being capable of realizing rapid acquisition of a large amount of region-level image-text data labeling results on the basis of ensuring that the determined region-level image-text labeling has high accuracy and reliability.

[0017] Other features and aspects of the present disclosure will become apparent from the following detailed description of exemplary embodiments, taken in conjunction with the accompanying drawings. BRIEF DESCRIPTION OF DRAWINGS

[0018] The accompanying drawings, which are incorporated in and constitute a part of the specification, illustrate exemplary embodiments, features, and aspects of the present disclosure and serve to explain the principles of the present disclosure.

[0019] FIG. 1 shows a flowchart of a multi-modal general perception model training method according to an embodiment of the present disclosure.

[0020] FIG. 2 shows a flowchart of a labeling method according to an embodiment of the present disclosure.

[0021] FIG. 3 shows a block diagram of a multi-modal general perception model training device according to an embodiment of the present disclosure.

[0022] FIG. 4 shows a block diagram of a labeling device according to an embodiment of the present disclosure.

[0023] FIG. 5 shows a block diagram of an electronic device according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0024] Various exemplary embodiments, features, and aspects of the present disclosure will be explained in detail below with reference to the accompanying drawings. The same reference numerals in the drawings denote the same or similar elements. Although various aspects of the embodiments are illustrated in the drawings, the drawings are not necessarily drawn to scale unless specifically indicated.

[0025] The term "exemplary" is used herein to mean "serving as an example, instance, or illustration." Any implementation described herein as "exemplary" is not necessarily to be construed as preferred or advantageous over other implementations.

[0026] The term "and / or" used herein only means an association relationship of the associated objects, and means that there can be three relationships, for example, A and / or B can mean that A exists alone, A and B exist together, and B exists alone. In addition, the term "at least one" herein means any one of a plurality or any combination of at least two of a plurality, for example, including at least one of A, B, and C can mean including any one or more elements selected from the set consisting of A, B, and C.

[0027] In addition, in order to better illustrate the present disclosure, numerous specific details are given in the specific embodiments below. Those skilled in the art should understand that the present disclosure can also be implemented without some specific details. In some examples, methods, means, elements and circuits that are well known to those skilled in the art are not described in detail in order to highlight the main idea of the present disclosure.

[0028] Large language models (LLMs) have strong reasoning and text understanding capabilities, and can maintain good generalization ability for long-tail distribution user instructions, and have wide application prospects. However, the LLMs commonly used in the prior art can only understand the text mode, and cannot perceive visual signals in the real world, resulting in limited application scenarios. Therefore, in the multi-modal field, a multi-modal general perception model is usually constructed based on the LLM to expand the perception boundary of the LLM, so that the LLM can perceive visual signals.

[0029] However, the common multi-modal general perception model in the prior art can only understand the input target image as a whole, cannot understand the "pointer instruction" input by the user, that is, cannot perform high-fineness perception and question answering on the target region and / or target object selected by the user in the target image, resulting in that when the user interacts with the artificial intelligence system based on the multi-modal general perception model, the user needs to emphasize the focus of the artificial intelligence system through a large amount of language description to achieve high-fineness perception and question answering, increasing the difficulty of the user interacting with the artificial intelligence system.

[0030] One of the reasons why the common multi-modal general perception model in the prior art cannot understand the "pointer instruction" input by the user is that there is a lack of region-level image-text data annotation results. In the prior art, the region-level image-text data annotation results are usually obtained through manual annotation. Specifically, the annotator needs to construct related questions according to the information of the sample image, then answers the questions to determine the corresponding answers, to generate the question-answer pair information corresponding to the sample image, and then determine the region-level image-text data annotation result corresponding to the sample image. Such an annotation process needs to consume a long working time. Moreover, the labor cost of manual annotation is usually settled in "words", resulting in high annotation cost. Therefore, in the prior art, it is difficult to quickly and effectively construct a large number of region-level image-text data annotation results.

[0031] Therefore, the embodiment of the present disclosure provides a multi-modal general perception model training method, which can simplify the process of region-level image-text annotation, reduce manual participation, realize efficient automatic annotation of a target image through a trained multi-modal general perception model, determine a region-level image-text data annotation result with high accuracy and reliability, reduce the cost of region-level image-text annotation, and improve the efficiency of region-level image-text annotation. The multi-modal general perception model training method provided by the embodiment of the present disclosure will be described in detail below.

[0032] FIG. 1 shows a flowchart of a multi-modal general perception model training method according to an embodiment of the present disclosure. As shown in FIG. 1, the multi-modal general perception model training method can be executed by an electronic device such as a terminal device or a server. The terminal device can be a user equipment (UE), a mobile device, a user terminal, a terminal, a cellular phone, a cordless phone, a personal digital assistant (PDA), a handheld device, a computing device, a vehicle-mounted device, a wearable device, etc. The multi-modal general perception model training method can be realized by a processor calling computer-readable instructions stored in a memory. Alternatively, the multi-modal general perception model training method can be executed by a server. As shown in FIG. 1, the multi-modal general perception model training method includes:

[0033] In step S11, a plurality of sample images are determined, and each sample image is labeled to determine a plurality of initial region-level image-text data labeling results corresponding to each sample image.

[0034] Specifically, any one sample image can include at least one target object; any one initial region-level image-text data labeling result corresponding to any one sample image can represent a labeling result of any one target object in the sample image. Wherein, the target object can be flexibly set according to actual use requirements, for example, the target object can include any one specified image region and / or object of the sample image, etc., and the present disclosure does not make specific limitation thereto; the specific content of the initial region-level image-text data labeling result can be flexibly set according to actual use requirements, for example, it can include region-level image data corresponding to any one target object in the sample image, and text data such as candidate semantic information, question and answer pair information and description information corresponding to any one target object, and the present disclosure does not make specific limitation thereto.

[0035] For any one sample image, the specific number of initial region-level image-text data labeling results corresponding to the sample image depends on the number of target objects included in the sample image, and the present disclosure does not make specific limitation thereto.

[0036] Hereinafter, the specific process of labeling each sample image to determine a plurality of initial region-level image-text data labeling results corresponding to each sample image will be described in detail in conjunction with possible implementation modes of the present disclosure, which will not be repeated here.

[0037] In step S12, the plurality of initial region-level image-text data labeling results corresponding to each sample image are corrected to determine a plurality of corrected labeling results corresponding to each sample image.

[0038] For any one sample image, the plurality of initial region-level image-text data labeling results corresponding to the sample image can be corrected to make the corrected labeling result corresponding to the sample image have high accuracy and accurately reflect the characteristics and state of each target object in the sample image.

[0039] Hereinafter, the specific process of correcting the plurality of initial region-level image-text data labeling results corresponding to each sample image to determine a plurality of corrected labeling results corresponding to each sample image will be described in detail in conjunction with possible implementation modes of the present disclosure, which will not be repeated here.

[0040] In step S13, the multi-modal general perception model is trained based on the plurality of corrected annotation results corresponding to each sample image, to obtain a trained multi-modal general perception model. The trained multi-modal general perception model is used to annotate a target image and determine a plurality of region-level image-text data annotation results corresponding to the target image. Any one of the region-level image-text data annotation results corresponding to the target image represents an individual annotation result of any one target object in the target image.

[0041] The multi-modal general perception model is trained by using the corrected annotation results with high accuracy, so that the trained multi-modal general perception model can individually annotate any one target object in an input target image, and the accuracy and reliability of the obtained region-level image-text data annotation results are improved. The specific manner of training the multi-modal general perception model based on the plurality of corrected annotation results corresponding to each sample image can refer to the implementation manner in the related art, and the present disclosure does not make a specific limitation thereto.

[0042] The specific form and structure of the multi-modal general perception model can be flexibly set according to actual use requirements, and the present disclosure does not make a specific limitation thereto.

[0043] In an example, the multi-modal general perception model can be a unified model supporting both discriminative tasks and generative tasks. The structure of the model can include a visual encoder, a visual-language connector, and a language model decoder.

[0044] The visual-language connector can be constructed based on a Q-Former structure and introduce a Region of Interest Align (RoI Align) operation to improve the understanding ability of the visual-language connector to regional information. Specifically, the RoI Align operation can extract corresponding regional features from the sample image according to preset position information, and input these regional features as query tokens to the Q-Former structure. These regional features as query tokens can be input to the language model decoder together with the randomly initialized extracted global features in the Q-Former structure for information decoding, to perform the discriminative task and / or the generative task of the multi-modal general perception model.

[0045] In the following, the specific process of training the multi-modal general perception model based on the plurality of corrected annotation results corresponding to each sample image to obtain the trained multi-modal general perception model will be described in detail in combination with possible implementation manners of the present disclosure, which will not be repeated here.

[0046] In the embodiments of the present disclosure, by determining a plurality of sample images and respectively labeling each sample image, a plurality of initial region-level image-text data labeling results corresponding to each sample image can be determined; by correcting the plurality of initial region-level image-text data labeling results corresponding to each sample image, a plurality of corrected labeling results corresponding to each sample image can be determined, thereby improving the accuracy of the corrected labeling results, so that any one of the corrected labeling results can accurately describe the characteristics and state of the target object corresponding to the corrected labeling result; based on the plurality of corrected labeling results corresponding to each sample image, the multi-modal general perception model is trained, and a trained multi-modal general perception model can be obtained, and the trained multi-modal general perception model can be used to label a target image and determine a plurality of region-level image-text data labeling results corresponding to the target image, any one of the region-level image-text data labeling results corresponding to the target image representing an individual labeling result of any one target object in the target image, thereby simplifying the process of region-level image-text labeling, reducing the participation of manual operation in the labeling process, reducing the workload and cost of region-level image-text labeling, improving the work efficiency of region-level image-text labeling, and realizing the rapid acquisition of a large amount of region-level image-text data labeling results on the basis of ensuring that the determined region-level image-text labeling has high accuracy and reliability.

[0047] In a possible implementation, respectively labeling each sample image to determine a plurality of initial region-level image-text data labeling results corresponding to each sample image includes: for any one sample image, determining position information corresponding to each target object in the sample image according to a position recognition model; for any one target object in the sample image, determining candidate semantic information corresponding to the target object according to a feature extraction model; for any one target object in the sample image, pairing the position information and the candidate semantic information corresponding to the target object according to an image-text alignment model to obtain pairing information corresponding to the target object; for any one target object in the sample image, determining question-answer pair information and description information corresponding to the target object according to the pairing information corresponding to the target object, wherein the question-answer pair information corresponding to the target object represents a question and an answer related to the target object, and the description information corresponding to the target object is used to describe the characteristics and state of the target object; and determining a plurality of initial region-level image-text data labeling results corresponding to the sample image according to the pairing information, the question-answer pair information, and the description information corresponding to each target object in the sample image.

[0048] In order to reduce the participation of manual operation in the region-level image-text labeling process and reduce the workload and cost of labeling, the embodiments of the present disclosure comprehensively utilize a plurality of different neural network models to preliminarily label a sample image in a region-level image-text manner to obtain an initial region-level image-text data labeling result.

[0049] Specifically, for any one sample image, the position recognition model can be used to locate each target object in the sample image, and the position information corresponding to each target object in the sample image is determined respectively. The position information corresponding to any one target object in the sample image can be used to indicate the position of the target object in the sample image. The specific form of the position information can be flexibly set according to actual use requirements, for example, it can be the coordinates of the center point of the target object in the sample image, or the corresponding indication box of the target object in the sample image, and the present disclosure does not make specific limitations thereto. The specific form and structure of the position recognition model can refer to the implementation manner in the related art, and the present disclosure does not make specific limitations thereto.

[0050] Using the feature extraction model, the features of each target object in the input sample image can be extracted, and the corresponding entity noun is outputted, the features of each target object are summarized, and the preset semantic category is combined to form the candidate semantic information corresponding to each target object.

[0051] The preset semantic category can be used to summarize the basic information of any target object in the sample image. The specific content of the semantic category can be flexibly set according to actual use requirements, and the present disclosure does not make specific limitations thereto.

[0052] In an example, when the sample image is a half-length portrait of an adult male wearing a blue shirt with a smiling expression, and the target object is the photographed person in the sample image, the semantic category corresponding to the target object can be set as “human” and the like.

[0053] For any one sample image, the candidate semantic information corresponding to any one target object in the sample image can be used to summarize the features of the target object. The specific content of the candidate semantic information can be flexibly set according to actual use requirements, and the present disclosure does not make specific limitations thereto. The specific number of candidate semantic information can be flexibly set according to actual use requirements, for example, the number of candidate semantic information corresponding to each target object can be set to 5, and the like, and the present disclosure does not make specific limitations thereto.

[0054] Taking the above sample image as a half-length portrait of an adult male wearing a blue shirt with a smiling expression, and the target object as the photographed person in the sample image as an example. In combination with the aforementioned semantic category, the number of candidate semantic information corresponding to the target object can be set to 4, including word groups describing the gender, race, clothing, expression and action of the photographed person, which can be man, white person, blue shirt, and smile.

[0055] The specific form and structure of the feature extraction model can refer to the implementation manner in the related art, and the present disclosure does not make specific limitations thereto.

[0056] For any one target object in the sample image, the position information and the candidate semantic information corresponding to the target object can be paired through the image-text alignment model to obtain pairing information corresponding to the target object, so as to specifically indicate that each candidate semantic information corresponds to a target object in the sample image, and the initial region-level image-text data annotation result determined subsequently can meet the use demand of the "pointer instruction". The specific form and structure of the image-text alignment model can refer to the implementation manner in the related art, and the present disclosure does not make a specific limitation in this regard.

[0057] For any one target object in the sample image, the pairing information corresponding to the target object can be used to determine the question-answer pair information and the description information corresponding to the target object. The specific manner of determining the question-answer pair information and the description information corresponding to the target object can refer to the implementation manner in the related art, for example, a multi-modal model in the prior art can be used to process the pairing information corresponding to the target object to determine the question-answer pair information and the description information corresponding to the target object, and the present disclosure does not make a specific limitation in this regard.

[0058] The question-answer pair information corresponding to any one target object can represent a question and an answer related to the target object. The specific content and quantity of the question-answer pair information corresponding to any one target object can be flexibly set according to actual use requirements, and the present disclosure does not make a specific limitation in this regard.

[0059] Taking the sample image as an example, the sample image is a half-length portrait of an adult male wearing a blue shirt with a smiling expression, and the target object is the person being photographed in the sample image. The question-answer pair information corresponding to the target object can include the question "What is the gender of the target object?" and the answer "The gender of the target object is male".

[0060] The description information corresponding to any one target object can be used to describe the characteristics and state of the target object. The specific content of the description information corresponding to any one target object can be flexibly set according to actual use requirements, and the present disclosure does not make a specific limitation in this regard.

[0061] Taking the sample image as an example, the sample image is a half-length portrait of an adult male wearing a blue shirt with a smiling expression, and the target object is the person being photographed in the sample image. The description information corresponding to the target object can include that the target object is a white male wearing a blue shirt and smiling, and the like.

[0062] For any one sample image, after the pairing information, the question-answer pair information and the description information corresponding to each target object in the sample image are determined, a plurality of initial region-level image-text data annotation results corresponding to the sample image can be further determined to complete the region-level image-text annotation of the sample image.

[0063] Further, any one of the region-level image and text data annotation results corresponding to any one of the sample images can also include region-level image data determined according to the pairing information. The region-level image data here can represent a cropped image including a target object corresponding to the region-level image and text data annotation result, and can intuitively reflect the target object corresponding to the region-level image and text data annotation result in the sample image.

[0064] Through the above process, the annotation process of the sample images can be simplified by comprehensively utilizing different neural network models, preliminary region-level image and text annotation of the sample images can be realized, a large number of initial region-level image and text data annotation results can be obtained, work costs can be reduced, and annotation efficiency can be improved.

[0065] In a possible implementation, correcting the multiple initial region-level image and text data annotation results corresponding to each sample image to determine multiple corrected annotation results corresponding to each sample image includes: dividing all the initial region-level image and text data annotation results corresponding to the multiple sample images into a first sample group and a second sample group, wherein the number of the initial region-level image and text data annotation results included in the first sample group is greater than the number of the initial region-level image and text data annotation results included in the second sample group; and correcting each initial region-level image and text data annotation result included in the second sample group respectively to determine a corrected second sample group.

[0066] The accuracy of the initial region-level image and text data annotation results determined through the foregoing process can be affected by the performance of the multiple neural network models used, and errors can exist. Therefore, in order to further improve the accuracy of the initial region-level image and text data annotation results and ensure that the trained multi-modal general perception model has high accuracy and reliability, it is necessary to check and correct the multiple initial region-level image and text data annotation results corresponding to each sample image.

[0067] Since the number of all the initial region-level image and text data annotation results corresponding to the multiple sample images can be relatively large, checking and correcting one by one can consume a large amount of work costs and time. Therefore, in order to improve the annotation efficiency, only part of the initial region-level image and text data annotation results can be checked and corrected in the embodiments of the present disclosure.

[0068] Specifically, all initial region-level image-text data annotation results corresponding to the plurality of sample images can be divided into a first sample group and a second sample group. The specific number of initial region-level image-text data annotation results included in the first sample group and the specific number of initial region-level image-text data annotation results included in the second sample group can be flexibly set according to actual use requirements, as long as the number of initial region-level image-text data annotation results included in the first sample group is greater than the number of initial region-level image-text data annotation results included in the second sample group, and the present disclosure does not make specific limitations thereto.

[0069] By respectively checking and correcting each initial region-level image-text data annotation result included in the second sample group, a corrected second sample group with a higher accuracy rate can be determined. Using the corrected second sample group, the multi-modal general perception model can be fine-tuned to improve the accuracy and reliability of the multi-modal general perception model.

[0070] The specific manner of respectively correcting each initial region-level image-text data annotation result included in the second sample group can refer to the implementation manner in the related art, for example, the checking and correction can be performed manually, and the present disclosure does not make specific limitations thereto.

[0071] In one possible implementation manner, respectively correcting each initial region-level image-text data annotation result included in the second sample group to determine the corrected second sample group includes: deleting incorrect candidate semantic information in each initial region-level image-text data annotation result included in the second sample group to obtain a second sample group after semantic checking; adjusting incorrect question and answer pair information in each initial region-level image-text data annotation result included in the second sample group after semantic checking to determine the corrected second sample group.

[0072] The correction of each initial region-level image-text data annotation result included in the second sample group can include semantic information correction and question and answer pair information correction. The semantic information correction is used to correct the candidate semantic information included in any one initial region-level image-text data annotation result in the second sample group. Specifically, for any one initial region-level image-text data annotation result in the second sample group, all candidate semantic information included in the initial region-level image-text data annotation result can be checked to determine whether each candidate semantic information correctly summarizes the characteristics of the corresponding target object, and the incorrect candidate semantic information can be deleted.

[0073] The question and answer pair information correction is used for correcting answers of question and answer pair information included in any one initial regional level image-text data labeling result in the second sample group. Specifically, for any one initial regional level image-text data labeling result in the second sample group, all question and answer pair information included in the initial regional level image-text data labeling result can be checked to determine whether the answer of each question and answer pair information is correct, and the incorrect answer is adjusted.

[0074] In a possible implementation, for any one question and answer pair information in any one initial regional level image-text data labeling result, when the answer of the question and answer pair information is correct, the question and answer pair information can be marked as "answer correct", and the question and answer pair information is retained.

[0075] In a possible implementation, for any one question and answer pair information in any one initial regional level image-text data labeling result, when the answer of the question and answer pair information cannot be directly determined according to the sample image, the question and answer pair information can be marked as "unanswerable", and the answer of the question and answer pair information is adjusted to a preset template.

[0076] The preset template is used to indicate that the question of the question and answer pair information cannot be answered, and the specific content of the preset template can be flexibly set according to actual use requirements, which is not limited in the present disclosure.

[0077] Taking the above sample image as an example, the sample image is a half-length portrait of an adult male wearing a blue shirt and smiling, and the target object is the photographed person in the sample image. When the question of a question and answer pair information is "what is the occupation of the target object?", since the answer to the question cannot be determined according to the sample image, the question and answer pair information can be marked as "unanswerable", and the answer of the question and answer pair information is adjusted to the preset template "the answer cannot be determined based on the existing information".

[0078] In a possible implementation, for any one question and answer pair information in any one initial regional level image-text data labeling result, when the answer of the question and answer pair information is incorrect, the question and answer pair information can be marked as "answer incorrect", and the answer of the question and answer pair information is corrected to give a correct answer.

[0079] Taking the above sample image as an example, the sample image is a half-length portrait of an adult male wearing a blue shirt and smiling, and the target object is the photographed person in the sample image. When the question of a question and answer pair information is "what is the gender of the target object?", and the answer to the question is "female", it can be determined that the answer of the question and answer pair information is incorrect, at this time, the question and answer pair information can be marked as "answer incorrect", and the answer of the question and answer pair information is corrected to "male".

[0080] Through the above process, the corrected label results of each initial region-level graphic and text data included in the second sample set can be corrected, and a corrected second sample set is determined, so as to obtain a corrected label result with higher accuracy. By using the corrected second sample set, the multi-modal general perception model can be fine-tuned to improve the accuracy and reliability of the multi-modal general perception model.

[0081] In a possible implementation, the multi-modal general perception model is trained based on the plurality of corrected label results corresponding to each sample image, and a trained multi-modal general perception model is obtained, including: pre-training the multi-modal general perception model according to the first sample set to obtain a pre-trained multi-modal general perception model; and fine-tuning the pre-trained multi-modal general perception model according to the corrected second sample set to determine the trained multi-modal general perception model.

[0082] Based on the first sample set including more initial region-level graphic and text data label results, the multi-modal general perception model can be pre-trained to quickly obtain a pre-trained multi-modal general perception model capable of realizing region-level graphic and text labeling. Based on the corrected second sample set with higher accuracy, the pre-trained multi-modal general perception model can be fine-tuned to improve the accuracy and reliability of the trained multi-modal general perception model in region-level graphic and text labeling.

[0083] The specific manner of pre-training the multi-modal general perception model and the specific manner of fine-tuning the pre-trained multi-modal general perception model can refer to the implementation in the related art, and the present disclosure does not make specific limitations thereto.

[0084] Through the trained multi-modal general perception model, the target image can be automatically and efficiently labeled, and a plurality of region-level graphic and text data label results corresponding to the target image can be determined, which can simplify the process of region-level graphic and text labeling of the target image, reduce human participation, improve the working efficiency of region-level graphic and text labeling, reduce the working cost, and realize large-scale acquisition of region-level graphic and text data label results.

[0085] In the embodiments of the present disclosure, by respectively labeling each sample image, the plurality of initial region-level image-text data labeling results corresponding to each sample image can be determined; by correcting the plurality of initial region-level image-text data labeling results corresponding to each sample image, the plurality of corrected labeling results corresponding to each sample image can be determined, thereby improving the accuracy of the corrected labeling results, so that any one of the corrected labeling results can accurately describe the features and states of the target object corresponding to the corrected labeling result; based on the plurality of corrected labeling results corresponding to each sample image, the multi-modal general perception model is trained, and the trained multi-modal general perception model can be obtained, and the trained multi-modal general perception model can be used to label the target image and determine the plurality of region-level image-text data labeling results corresponding to the target image, any one of the region-level image-text data labeling results corresponding to the target image represents an individual labeling result of any one target object in the target image, thereby simplifying the process of region-level image-text labeling, reducing the participation of manual operation in the labeling process, reducing the workload and cost of region-level image-text labeling, improving the work efficiency of region-level image-text labeling, and on the basis of ensuring that the determined region-level image-text labeling has high accuracy and reliability, a large amount of region-level image-text data labeling results can be quickly obtained.

[0086] The present disclosure also provides a labeling method. FIG. 2 shows a flowchart of a labeling method according to an embodiment of the present disclosure. The labeling method can be performed by an electronic device such as a terminal device or a server, and the terminal device can be a user equipment (UE), a mobile device, a user terminal, a terminal, a cellular phone, a cordless phone, a personal digital assistant (PDA), a handheld device, a computing device, a vehicle-mounted device, a wearable device, etc. The labeling method can be implemented by a processor invoking computer-readable instructions stored in a memory. Alternatively, the labeling method can be performed by a server. As shown in FIG. 2, the labeling method includes:

[0087] In step S21, a target image is determined, wherein the target image includes at least one target object.

[0088] The target object herein can represent any one specified image region and / or object in the target image, and can be flexibly set according to actual use requirements, which is not limited in the present disclosure.

[0089] The specific manner of determining the target image can refer to the implementation in the related art, which is not limited in the present disclosure.

[0090] In step S22, the target image is automatically labeled according to the multi-modal general perception model to determine a plurality of region-level image-text data labeling results corresponding to the target image. Any one of the region-level image-text data labeling results corresponding to the target image represents a separate labeling result of any one of the target objects in the target image.

[0091] The multi-modal general perception model herein can represent a model obtained according to the multi-modal general perception model training method described above. Inputting the target image into the multi-modal general perception model can achieve automatic labeling of the target image and determine a plurality of region-level image-text data labeling results corresponding to the target image.

[0092] Any one of the region-level image-text data labeling results corresponding to the target image can represent a separate labeling result of any one of the target objects in the target image. The specific content of the region-level image-text data labeling result can be flexibly set according to actual use requirements. For example, it can include region-level image data corresponding to any one of the target objects in the sample image, and text data such as candidate semantic information, question and answer pair information, and description information corresponding to any one of the target objects. The present disclosure does not make specific limitations on this.

[0093] In the embodiments of the present disclosure, the target image is determined, wherein the target image includes at least one target object; and the target image can be labeled according to the multi-modal general perception model to determine a plurality of region-level image-text data labeling results corresponding to the target image. Any one of the region-level image-text data labeling results corresponding to the target image represents a separate labeling result of any one of the target objects in the target image. Thus, the process of region-level image-text labeling is simplified, the participation of manual operation in the labeling process is reduced, the workload and cost of region-level image-text labeling are reduced, the work efficiency of region-level image-text labeling is improved, and a large amount of region-level image-text data labeling results can be quickly obtained on the basis of ensuring that the determined region-level image-text labeling has high accuracy and reliability.

[0094] It can be understood that the above-mentioned various method embodiments of the present disclosure can be combined with each other to form combined embodiments without violating the principle logic. Limited by the length of the present disclosure, the present disclosure will not be described again. Those skilled in the art can understand that the specific execution order of each step in the above-mentioned method of the specific embodiment should be determined according to its function and possible internal logic.

[0095] In addition, the disclosure also provides a multi-modal general perception model training device, a labeling device, an electronic device, and a non-volatile computer readable storage medium and a program, which can be used to implement any one of the multi-modal general perception model training and / or labeling methods provided by the disclosure. The corresponding technical solutions and descriptions and the corresponding records in the method part are not repeated.

[0096] FIG. 3 shows a block diagram of a multi-modal general perception model training device according to an embodiment of the disclosure. As shown in FIG. 3, the device 300 includes:

[0097] An initial labeling module 301 is configured to determine a plurality of sample images, and label each sample image respectively to determine a plurality of initial region-level image-text data labeling results corresponding to each sample image.

[0098] A correction module 302 is configured to correct the plurality of initial region-level image-text data labeling results corresponding to each sample image to determine a plurality of corrected labeling results corresponding to each sample image.

[0099] A model training module 303 is configured to train the multi-modal general perception model based on the plurality of corrected labeling results corresponding to each sample image to obtain a trained multi-modal general perception model, wherein the trained multi-modal general perception model is configured to label a target image to determine a plurality of region-level image-text data labeling results corresponding to the target image, and any one of the region-level image-text data labeling results corresponding to the target image represents an individual labeling result of any one target object in the target image.

[0100] In a possible implementation, the initial labeling module 301 is specifically configured to: for any one sample image, determine position information corresponding to each target object in the sample image according to a position recognition model; for any one target object in the sample image, determine candidate semantic information corresponding to the target object according to a feature extraction model; for any one target object in the sample image, pair the position information and the candidate semantic information corresponding to the target object according to an image-text alignment model to obtain pairing information corresponding to the target object; for any one target object in the sample image, determine question-answer pair information and description information corresponding to the target object according to the pairing information corresponding to the target object, wherein the question-answer pair information corresponding to the target object represents a question and an answer related to the target object, and the description information corresponding to the target object is used to describe the features and the state of the target object; and determine a plurality of initial region-level image-text data labeling results corresponding to the sample image according to the pairing information, the question-answer pair information, and the description information corresponding to each target object in the sample image.

[0101] In a possible implementation, the correction module 302 is specifically configured to: divide all initial region-level image-text data annotation results corresponding to the plurality of sample image pairs into a first sample group and a second sample group, where the number of initial region-level image-text data annotation results included in the first sample group is greater than the number of initial region-level image-text data annotation results included in the second sample group; and correct each initial region-level image-text data annotation result included in the second sample group respectively to determine a second sample group after correction.

[0102] In a possible implementation, the correction module 302 is specifically configured to: delete incorrect candidate semantic information in each initial region-level image-text data annotation result included in the second sample group to obtain a second sample group after semantic verification; and adjust incorrect question-answer pair information in each initial region-level image-text data annotation result included in the second sample group after semantic verification to determine a second sample group after correction.

[0103] In a possible implementation, the model training module 303 is specifically configured to: pre-train the multi-modal general perception model according to the first sample group to obtain a pre-trained multi-modal general perception model; and fine-tune the pre-trained multi-modal general perception model according to the second sample group after correction to determine a trained multi-modal general perception model.

[0104] FIG. 4 shows a block diagram of an annotation apparatus according to an embodiment of the present disclosure. As shown in FIG. 4, the apparatus 400 includes:

[0105] The image acquisition module 401 is configured to determine a target image, where the target image includes at least one target object.

[0106] The annotation module 402 is configured to automatically annotate the target image according to a multi-modal general perception model to determine a plurality of region-level image-text data annotation results corresponding to the target image, where the multi-modal general perception model is trained by the method described above, and any one region-level image-text data annotation result corresponding to the target image represents an individual annotation result of any one target object in the target image.

[0107] In some embodiments, the apparatus provided by the embodiments of the present disclosure has functions or includes modules that can be used to perform the methods described in the above method embodiments, and the specific implementation can refer to the description of the above method embodiments. For brevity, they will not be described here.

[0108] The embodiments of the present disclosure also provide a computer-readable storage medium having computer program instructions stored therein, where the computer program instructions are executed by a processor to implement the above method. The computer-readable storage medium can be a volatile or non-volatile computer-readable storage medium.

[0109] The embodiments of the present disclosure further provide an electronic device, comprising: a processor; a memory for storing processor-executable instructions; wherein the processor is configured to implement the above method when executing the instructions stored in the memory.

[0110] FIG. 5 shows a block diagram of an electronic device according to an embodiment of the present disclosure. For example, the electronic device 1900 can be provided as a server or a terminal device. Referring to FIG. 5, the apparatus 1900 includes a processing component 1922, which further includes one or more processors, and a memory resource represented by a memory 1932 for storing instructions executable by the processing component 1922, such as an application program. The application program stored in the memory 1932 can include one or more than one module each corresponding to a set of instructions. In addition, the processing component 1922 is configured to execute the instructions to perform the above method.

[0111] The apparatus 1900 can further include a power supply component 1926 configured to perform power management of the apparatus 1900, a wired or wireless network interface 1950 configured to connect the apparatus 1900 to a network, and an input / output interface 1958 (I / O interface). The apparatus 1900 can operate based on an operating system stored in the memory 1932, such as Windows Server TM , Mac OS X TM , Unix TM , Linux TM , FreeBSD TM or the like.

[0112] In exemplary embodiments, a non-transitory computer readable storage medium, such as the memory 1932 including computer program instructions executable by the processing component 1922 of the apparatus 1900 to perform the above method is also provided.

[0113] The present disclosure can be a system, a method, and / or a computer program product. The computer program product can include a computer readable storage medium (or media) having computer readable program instructions thereon for causing a processor to carry out aspects of the present disclosure.

[0114] Computer readable storage media can be tangible storage media which can retain and store instructions for use by an instruction execution device. Computer readable storage media can be, for example, but is not limited to, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing. More specific examples (a non-exhaustive list) of computer readable storage media include the following: a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), a static random access memory (SRAM), a portable compact disc read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanically encoded device such as punch-cards or raised structures in a groove having instructions recorded thereon, and any suitable combination of the foregoing. A computer readable storage medium, as used herein, is not to be construed as being transitory signals per se, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through a waveguide or other transmission media (e.g., light pulses passing through a fiber-optic cable), or electrical signals transmitted through a wire.

[0115] Computer readable program instructions described herein can be downloaded to respective computing / processing devices from a computer readable storage medium or to an external computer or external storage device via a network, for example, the Internet, a local area network, a wide area network and / or a wireless network. The network can comprise copper transmission cables, optical transmission fibers, wireless transmission, routers, firewalls, switches, gateway computers and / or edge servers. A network adapter card or network interface in each computing / processing device receives computer readable program instructions from the network and forwards the computer readable program instructions for storage in a computer readable storage medium within the respective computing / processing device.

[0116] Computer readable program instructions for carrying out operations of the present disclosure can be assembler instructions, instruction-set-architecture (ISA) instructions, machine instructions, machine dependent instructions, microcode, firmware instructions, state-setting data, or either source code or object code written in any combination of one or more programming languages, including an object oriented programming language such as Smalltalk, C++ or the like and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The computer readable program instructions can execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the latter scenario, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection can be made to an external computer (for example, through the Internet using an Internet Service Provider). In some embodiments, electronic circuitry including, for example, programmable logic circuitry, field-programmable gate array (FPGA), or programmable logic array (PLA) can execute the computer readable program instructions by utilizing state information of the computer readable program instructions to personalize the electronic circuitry, in order to perform aspects of the present disclosure.

[0117] The computer readable program instructions can also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable apparatus or other device to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide processes for implementing the functions / acts specified in the flowchart and / or block diagram block or blocks.

[0118] These computer readable program instructions can be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions / acts specified in the flowchart and / or block diagram block or blocks. These computer readable program instructions can also be stored in a computer readable storage medium that can include random access memory (RAM), read only memory (ROM), electrically erasable programmable read only memory (EEPROM), flash memory or other data storage device. When the computer readable program instructions are loaded into the computer and other programmable data processing apparatus, a series of operational steps are implemented that provide processes such that the computer or other programmable apparatus provide processes for implementing the functions / acts specified in the flowchart and / or block diagram block or blocks.

[0119] The computer readable program instructions can also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus or other device to produce a computer implemented process such that the instructions which execute on the computer, other programmable data processing apparatus, or other device implement the functions / acts specified in the flowchart and / or block diagram block or blocks.

[0120] The flow diagrams and the block diagrams in the drawings are presented to illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each block in the flow diagrams and the block diagrams can represent a module, segment, or portion of instructions, which comprises one or more executable instructions for implementing the specified logic functions. In some alternative implementations, the functions noted in the blocks can occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks can sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams and / or flow diagrams, and combinations thereof, can be implemented by special purpose hardware-based systems that perform the specified functions or acts, or combinations of special purpose hardware and

[0121] Embodiments of the present disclosure have been described above, and the description is intended to be illustrative of the embodiments and not restrictive. Many modifications and variations of the described embodiments are possible and are within the scope of the disclosure. The selection of terms is intended to best describe the principles of the embodiments, practical application, or technical improvements in the art, or to enable others skilled in the art to understand the embodiments disclosed herein.

Claims

1. A training method for a multimodal general perception model, wherein, include: Multiple sample images are identified, and each sample image is labeled to determine the multiple initial region-level image and text data labeling results corresponding to each sample image; The initial region-level image and text data annotation results corresponding to each sample image are corrected to determine the multiple corrected annotation results corresponding to each sample image. Based on multiple corrected annotation results corresponding to each sample image, a multimodal general perception model is trained to obtain a trained multimodal general perception model. The trained multimodal general perception model is used to annotate target images and determine multiple regional-level image and text data annotation results corresponding to the target image. Any regional-level image and text data annotation result corresponding to the target image represents the annotation result of any target object in the target image.

2. The method according to claim 1, wherein, The step of annotating each sample image separately and determining multiple initial region-level image and text data annotation results corresponding to each sample image includes: For any given sample image, the location information corresponding to each target object in the sample image is determined according to the location recognition model; For any target object in the sample image, the candidate semantic information corresponding to the target object is determined according to the feature extraction model; For any target object in the sample image, the location information and candidate semantic information corresponding to the target object are paired according to the image-text alignment model to obtain the matching information corresponding to the target object. For any target object in the sample image, the question-answer pair information and description information corresponding to the target object are determined based on the pairing information corresponding to the target object. The question-answer pair information corresponding to the target object represents the questions and answers related to the target object, and the description information corresponding to the target object is used to describe the features and state of the target object. Based on the pairing information, question-answer pair information, and description information corresponding to each target object in the sample image, the initial regional-level image and text data annotation results corresponding to the sample image are determined.

3. The method according to claim 1 or 2, wherein, The step of correcting the multiple initial region-level image and text data annotation results corresponding to each sample image, and determining the multiple corrected annotation results corresponding to each sample image, includes: The initial region-level image and text data annotation results corresponding to the multiple sample images are divided into a first sample group and a second sample group, wherein the number of initial region-level image and text data annotation results included in the first sample group is greater than the number of initial region-level image and text data annotation results included in the second sample group. The annotation results of each initial region-level graphic data included in the second sample group are corrected to determine the corrected second sample group.

4. The method according to claim 3, wherein, The step of correcting the annotation results of each initial region-level graphic data included in the second sample group to determine the corrected second sample group includes: Delete erroneous candidate semantic information from each initial regional-level image and text data annotation result included in the second sample group to obtain the semantically validated second sample group; The corrected second sample group is determined by adjusting the erroneous question-answer pairs in each initial regional-level image and text data annotation result of the second sample group after semantic verification.

5. The method according to claim 3, wherein, The multimodal general perception model is trained based on multiple corrected annotation results corresponding to each sample image to obtain a trained multimodal general perception model, including: Based on the first sample group, the multimodal general perception model is pre-trained to obtain the pre-trained multimodal general perception model; Based on the corrected second sample group, the pre-trained multimodal general perception model is fine-tuned to determine the trained multimodal general perception model.

6. A labeling method, wherein, include: Determine a target image, wherein the target image includes at least one target object; The target image is automatically labeled according to the multimodal general perception model to determine multiple region-level image and text data labeling results corresponding to the target image. The multimodal general perception model is trained by the method described in any one of claims 1 to 5. Any region-level image and text data labeling result corresponding to the target image represents the labeling result of any target object in the target image.

7. A training device for a multimodal universal perception model, wherein, include: The initial annotation module is used to identify multiple sample images and annotate each sample image separately, thereby determining the initial region-level image and text data annotation results corresponding to each sample image; The correction module is used to correct the multiple initial region-level graphic data annotation results corresponding to each sample image, and to determine the multiple corrected annotation results corresponding to each sample image. The model training module is used to train the multimodal general perception model based on multiple corrected annotation results corresponding to each sample image, so as to obtain a trained multimodal general perception model. The trained multimodal general perception model is used to annotate the target image and determine multiple region-level image and text data annotation results corresponding to the target image. Any region-level image and text data annotation result corresponding to the target image represents the annotation result of any target object in the target image.

8. A labeling device, wherein, include: An image acquisition module is used to determine a target image, wherein the target image includes at least one target object; The annotation module is used to automatically annotate the target image according to the multimodal general perception model and determine multiple region-level image and text data annotation results corresponding to the target image. The multimodal general perception model is trained by the method described in any one of claims 1 to 5. Any region-level image and text data annotation result corresponding to the target image represents the annotation result of any target object in the target image.

9. An electronic device, wherein, include: processor; Memory used to store processor-executable instructions; The processor is configured to implement the method of any one of claims 1 to 6 when executing instructions stored in the memory.

10. A non-volatile computer-readable storage medium having computer program instructions stored thereon, wherein, When the computer program instructions are executed by the processor, they implement the method described in any one of claims 1 to 6.

Images