Multimodal data annotation method and electronic device

Abstract

The present application provides a multimodal data annotation method and an electronic device, the method comprising: first responding to a user's operation, displaying a first instruction text and multimedia resources corresponding to the multimodal data to be annotated on a display interface, then responding to the user's automatic annotation operation, based on the first instruction text and the multimodal data to be annotated, using at least one multimodal algorithm, generating at least one first response text, finally determining a first target response text based on the at least one generated first response text, and displaying the first target response text on the display interface, using the first instruction text, the multimodal data to be annotated, and the first target response text as a sample in a multimodal instruction data set, thereby realizing automatic generation of response text and automatic construction of a multimodal instruction data set, which can reduce annotation time and improve annotation efficiency compared to manual annotation of multimodal data.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a multimodal data annotation method and an electronic device. Background Art

[0002] Multimodal large models are cutting-edge technologies in the current field of artificial intelligence and have attracted widespread attention. In actual business applications, multimodal large models usually need to be fine-tuned based on multimodal instruction datasets. Manual labeling of multimodal data is an important step in building multimodal instruction datasets.

[0003] However, manual labeling of multimodal data is time-consuming and inefficient. Summary of the invention

[0004] The present application provides a multimodal data labeling method and electronic device to solve the problem that manual labeling of multimodal data is time-consuming and inefficient.

[0005] In a first aspect, the present application provides a multimodal data annotation method, comprising:

[0006] In response to the user's operation, displaying the first instruction text and the multimedia resources corresponding to the multimodal data to be annotated on the display interface;

[0007] In response to the user's automatic annotation operation, based on the first instruction text and the multimodal data to be annotated, at least one multimodal algorithm is used to generate at least one first response text;

[0008] If a first response text is generated, the first response text is used as the first target response text; if multiple first response texts are generated, the first target response text is obtained based on the similarity of the multiple first response texts or by fusing the multiple first response texts;

[0009] In a possible implementation, the method further includes:

[0010] Displaying a second instruction text in a preset template on the display interface;

[0011] The step of responding to the user's operation and displaying the first instruction text on the display interface includes:

[0012] In response to a user's selection operation, using the second instruction text as the first instruction text; or

[0013] In response to a first editing operation by the user, the second instruction text is edited to obtain the first instruction text; or

[0014] In response to the user's second editing operation, the first instruction text is generated.

[0015] In a possible implementation manner, after displaying the first target response text on the display interface, the method further includes:

[0016] In response to a third editing operation of the user, the first target response text is edited to obtain a second target response text;

[0017] The step of displaying the first target response text on the display interface and using the to-be-annotated multimodal data, the first instruction text and the first target response text as a multimodal instruction data set includes:

[0018] The second target response text is displayed on the display interface, and the multimodal data to be annotated, the first instruction text and the second target response text are used as a sample in the multimodal instruction data set.

[0019] In a possible implementation manner, after the display interface displays the first target response text, the method further includes:

[0020] In response to a user's marking operation on the multimedia resource, displaying a marked area on the display interface;

[0021] In response to a fourth editing operation of the user, generating a third instruction text, and displaying the third instruction text on the display interface;

[0022] Based on the multimodal data corresponding to the marked area and the third instruction text, at least one multimodal algorithm is used to generate at least one second response text;

[0023] If one second response text is generated, the second response text is used as the third target response text; if multiple second response texts are generated, the third target response text is obtained based on the similarity of the multiple second response texts or by fusing the multiple second response texts;

[0024] The third target response text is displayed on the display interface.

[0025] In a possible implementation manner, after displaying the third target response text on the display interface, the method further includes:

[0026] In response to a fifth editing operation of the user, the third target response text is edited to obtain a fourth target response text;

[0027] The fourth target response text is displayed on the display interface.

[0028] In a possible implementation, the method further includes:

[0029] The display interface shows some or all of the following:

[0030] Application scenarios, data types, template types, and model settings.

[0031] In a possible implementation, obtaining the first target response text based on the similarity of the plurality of first response texts includes:

[0032] Extracting text features of each first response text;

[0033] For any two first response texts, based on the text features of the two first response texts, calculating the similarity of the two first response texts;

[0034] Any one of the two first response texts with the greatest similarity is used as the first target response text.

[0035] In a possible implementation, obtaining the first target response text based on the similarity of the plurality of first response texts includes:

[0036] Extracting text features of each first response text;

[0037] For any two first response texts, based on the text features of the two first response texts, calculating the similarity of the two first response texts;

[0038] The two first response texts with the greatest similarity are merged to obtain the first target response text.

[0039] In a possible implementation, obtaining the first target response text based on the similarity of the plurality of first response texts includes:

[0040] Extracting image features of the multimodal data to be annotated and text features of each first response text;

[0041] Based on the image feature and the text feature of any one of the first response texts, calculating the similarity between the first response text and the multimodal data to be annotated;

[0042] The first response text corresponding to the maximum similarity among the calculated similarities is used as the first target response text.

[0043] In a second aspect, the present application provides an electronic device, including:

[0044] Memory for storing computer programs;

[0045] The processor is used to call the computer program stored in the memory and execute according to the obtained program:

[0046] In response to the user's automatic annotation operation, at least one first response text is generated based on the first instruction text and the multimodal data to be annotated, using at least one multimodal algorithm; if one first response text is generated, the first response text is used as the first target response text; if multiple first response texts are generated, the first target response text is obtained based on the similarity of the multiple first response texts or by fusing the multiple first response texts;

[0047] A display, used to respond to a user's operation, display the first instruction text and the multimedia resources corresponding to the multimodal data to be annotated on a display interface; and display the first target response text on the display interface.

[0048] The beneficial effects of the present invention are as follows:

[0049] The present application provides a multimodal data annotation method and electronic device, wherein the method includes, first responding to a user's operation, displaying a first instruction text and multimedia resources corresponding to the multimodal data to be annotated on a display interface, then responding to the user's automatic annotation operation, based on the first instruction text and the multimodal data to be annotated, using at least one multimodal algorithm, generating at least one first response text, and finally determining a first target response text based on the at least one generated first response text, and displaying the first target response text on the display interface, using the first instruction text, the multimodal data to be annotated, and the first target response text as a sample in a multimodal instruction data set, thereby realizing automatic generation of response text and automatic construction of a multimodal instruction data set, which can reduce annotation time and improve annotation efficiency compared to manual annotation of multimodal data. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0051] Figure 1 A schematic diagram of a process flow of a multimodal data annotation method provided in an embodiment of the present application;

[0052] Figure 2 A schematic diagram of the structure of a preset template content provided in an embodiment of the present application;

[0053] Figure 3 A schematic diagram of a display interface provided in an embodiment of the present application;

[0054] Figure 4A schematic diagram of another display interface provided in an embodiment of the present application;

[0055] Figure 5 A schematic diagram of a process for determining a first target response text provided in an embodiment of the present application;

[0056] Figure 6 A schematic diagram of another process for determining a first target response text provided in an embodiment of the present application;

[0057] Figure 7 A schematic diagram of another process for determining a first target response text provided in an embodiment of the present application;

[0058] Figure 8 A schematic diagram of another process for determining a first target response text provided in an embodiment of the present application;

[0059] Fig. 9 A schematic diagram of another process for determining a first target response text provided in an embodiment of the present application;

[0060] Fig.10 A schematic diagram of a sample structure of multimodal instruction data provided in an embodiment of the present application;

[0061] Fig.11 A schematic diagram of a sample structure of another multimodal instruction data provided in an embodiment of the present application;

[0062] Fig.12 A schematic diagram of a sample structure of another multimodal instruction data provided in an embodiment of the present application;

[0063] Fig.13 A schematic diagram of a display interface provided in an embodiment of the present application;

[0064] Fig.14 A schematic diagram of a sample structure of another multimodal instruction data provided in an embodiment of the present application;

[0065] Fig.15 A schematic diagram of another display interface provided in an embodiment of the present application;

[0066] Fig.16 A schematic diagram of a shortcut menu for a display interface provided in an embodiment of the present application;

[0067] Fig.17 A schematic diagram of another display interface provided in an embodiment of the present application;

[0068] Fig.18 A schematic diagram of another display interface shortcut menu provided in an embodiment of the present application;

[0069] Fig.19A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0070] In order to make the purpose, technical scheme and advantages of the present invention clearer, the present invention will be further described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0071] Multimodal data, including but not limited to image data, video data, audio data and text, can provide rich information. In order to obtain an accurate multimodal large model, a multimodal instruction dataset is usually used to fine-tune the multimodal large model. When constructing a multimodal instruction dataset, there are problems of long time consumption and low efficiency.

[0072] Based on the above technical problems, an embodiment of the present application provides a multimodal data annotation method and an electronic device, wherein the multimodal data annotation method includes: first responding to the user's operation, displaying a first instruction text and multimedia resources corresponding to the multimodal data to be annotated on a display interface, and then responding to the user's automatic annotation operation, based on the first instruction text and the multimodal data to be annotated, using at least one multimodal algorithm to generate at least one first response text, and finally determining a first target response text based on the at least one generated first response text, and displaying the first target response text on the display interface, using the first instruction text, the multimodal data to be annotated and the first target response text as a sample in a multimodal instruction data set, thereby realizing automatic generation of response text and automatic construction of a multimodal instruction data set. Compared with manual annotation of multimodal data, the annotation time can be reduced and the annotation efficiency can be improved.

[0073] A multimodal data annotation method and an electronic device provided in an embodiment of the present application are described in detail below in conjunction with the accompanying drawings.

[0074] like Figure 1 FIG. 1 is a flow chart of a multimodal data annotation method provided in an embodiment of the present application. The specific implementation process of the method is as follows:

[0075] S101: In response to a user operation, displaying a first instruction text and multimedia resources corresponding to the multimodal data to be annotated on a display interface;

[0076] S102: In response to the user's automatic annotation operation, based on the first instruction text and the multimodal data to be annotated, at least one multimodal algorithm is used to generate at least one first response text;

[0077] S103: if a first response text is generated, the first response text is used as the first target response text; if multiple first response texts are generated, the first target response text is obtained based on the similarity of the multiple first response texts or by fusing the multiple first response texts;

[0078] S104: Display the first target response text on the display interface.

[0079] In an embodiment of the present application, in response to a user's operation, a first instruction text and multimedia resources corresponding to the multimodal data to be annotated are displayed on a display interface, and then in response to the user's automatic annotation operation, at least one first response text is generated based on the first instruction text and the multimodal data to be annotated using at least one multimodal algorithm; if a first response text is generated, the first response text is used as the first target response text; if multiple first response texts are generated, the first target response text is obtained based on the similarity of the multiple first response texts or by fusing the multiple first response texts, and the first target response text is displayed on the display interface. Therefore, the problem of time-consuming and low efficiency of manual annotation in multimodal data annotation is solved.

[0080] The multimodal data to be annotated in the embodiment of the present application may be image data, video data, or other multimodal data. If the multimodal data to be annotated is image data, the multimedia resource displayed on the display interface is an image; if the multimodal data to be annotated is video data, the multimedia resource displayed on the display interface is a video.

[0081] In the specific implementation, first set the preset template, such as Figure 2 As shown, the preset template may include part or all of the application scenario, data type, template type, model settings, and operating instructions.

[0082] Among them, application scenarios may include smart transportation, smart public works, smart factories, smart cities, etc.; data types may include video-text multimodality and image-text multimodality; template types may include visual description, image-text matching, video-text matching, visual question-answering, etc.; model settings may include Qwen-vl, emu2, Intern-vl, etc.; operating instructions include instructions for multimodal data annotation methods.

[0083] It should be noted that the model in the mode setting, that is, the model called in the background, is also the multimodal algorithm in the embodiment of the present application.

[0084] After the preset template is set, the preset template is displayed on the display interface, such as Figure 3As shown, it is a schematic diagram of a display interface provided in an embodiment of the present application. A preset template is displayed on the display interface, and then in response to user operations, various options in the preset template are selected. For example, the application scenario selects "Smart Transportation", the data type selects "Image Text Multimodality", and the template type selects "Graphic Description".

[0085] The preset template may also include an instruction text library and a multimedia resource library. The user may select an instruction text from the instruction text library and a multimedia resource from the multimedia resource library. For example, Figure 4 As shown, the instruction text selected from the instruction text library is "Please describe the person in the picture", a picture is selected from the multimedia resource library, and then the instruction text and the picture are displayed on the display interface.

[0086] After selecting the instruction text and the picture, in response to the user's automatic annotation operation, the instruction text and the multimodal data to be annotated corresponding to the picture can be input into at least one selected multimodal algorithm to generate at least one first response text. Alternatively, based on the user's editing operation, the selected instruction text can be edited to obtain the first instruction text, and then the first instruction text and the multimodal data to be annotated corresponding to the selected multimedia resource can be input into at least one multimodal algorithm to generate at least one first response text.

[0087] For example, the second instruction text "Please describe the people in the picture" is displayed on the display interface. In response to the user's selection operation, the second instruction text "Please describe the people in the picture" is directly used as the first instruction text. Or in response to the user's first editing operation, the second instruction text "Please describe the people in the picture" is edited to obtain "Please describe the little girl in the picture". After obtaining the edited instruction text, the instruction text is displayed on the display interface.

[0088] In one embodiment, the first instruction text may be directly generated based on a second editing operation of the user, in addition to being selected from an instruction text library or being edited from an instruction text selected from the instruction text library.

[0089] For example, Figure 4 As shown, after the user selects a picture as the multimodal data to be annotated, he edits an instruction text "Please describe the little girl in the picture", and displays the edited instruction text as the first instruction text on the display interface.

[0090] If you edit the selected instruction text or generate an instruction text, you can add the instruction text to the instruction text library.

[0091] In one embodiment, based on the first instruction text and the multimodal data to be annotated, at least one multimodal algorithm is used to generate at least one first response text. Specifically, text features of each first response text can be extracted; for any two first response texts, the similarity of the two first response texts is calculated based on the text features of the two first response texts; and any one of the two first response texts with the greatest similarity is used as the first target response text.

[0092] Specifically, Figure 5 As shown, it is a schematic diagram of a process of determining a first target response text provided in an embodiment of the present application:

[0093] S501: Determine whether the number of first response texts is multiple, if so, execute S502, otherwise end;

[0094] S502: extracting the text feature An of each first response text and calculating the similarity Sn between any two first response texts;

[0095] S503: Select the first maximum similarity;

[0096] S504: taking any one of the two first response texts corresponding to the maximum similarity as the first target response text.

[0097] For example, if the first instruction text is: "Is the person in the upper left corner of the picture smoking?", the first response texts generated are:

[0098] First response text T1: “Yes”;

[0099] First response text T2: “No”;

[0100] First response text T3: "Yes".

[0101] After extracting text feature A1, text feature A2 and text feature A3 respectively, the first response text similarity is calculated, wherein, if the two responses are similar, the first target response text is “yes”.

[0102] In another embodiment, based on the first instruction text and the multimodal data to be annotated, at least one multimodal algorithm is used to generate at least one first response text. Specifically, text features of each first response text are extracted; for any two first response texts, the similarity of the two first response texts is calculated based on the text features of the two first response texts; the two first response texts with the greatest similarity are fused to obtain a first target response text.

[0103] Specifically, Figure 6As shown, it is a schematic diagram of another process of determining the first target response text provided in an embodiment of the present application:

[0104] S601: Determine whether the number of first response texts is multiple, if so, execute S302, otherwise end;

[0105] S602: extracting the text feature An of each first response text and calculating the similarity Sn between any two first response texts;

[0106] S603: Select the first maximum similarity;

[0107] S604: The first response text obtained by merging the two first response texts corresponding to the maximum similarity is used as the first target response text.

[0108] For example, three first response texts are generated:

[0109] First response text T1: "In the upper left corner, there is a man wearing a light-colored top and dark pants. He is standing next to a wall, holding a mobile phone in his hand and seems to be making a call."

[0110] First responder text T2: "The person on the upper left, wearing a light-colored top and blue jeans, is standing by the wall, holding the phone with both hands, close to his ear, and appears to be on a call."

[0111] First response text T3: “In the upper left corner of the image, a young man is seen, he is wearing casual clothes and is talking on a mobile phone.

[0112] After extracting text features A1, A2, and A3, respectively, calculate

[0113] The first similarity S1 between T1 and T2 is 0.85;

[0114] The second similarity S2 between T1 and T2 is 0.70;

[0115] The third similarity S3 between T2 and T3 is 0.75.

[0116] After comparison, the first similarity S1 is the highest, and the first response text T1 and the first response text T2 are merged as the first target response text. The first target response text is: "In the upper left corner, there is a man wearing a light-colored top and dark blue pants. He is standing next to a wall, holding a mobile phone with both hands close to his ear, and seems to be making a call."

[0117] like Figure 7 As shown, it is a schematic diagram of a process of determining a first target response text provided in the above embodiment; the details are as follows:

[0118] S701: Generate three first response texts T1, T2, T3;

[0119] S702: extracting text features A1, A2, A3 of three first response texts;

[0120] S703: Calculate a first similarity S1 between T1 and T2, a second similarity S2 between T1 and T2, and a third similarity S3 between T2 and T3;

[0121] S704: Select the maximum similarity from S1, S2, and S3;

[0122] S705: Taking the response text corresponding to the maximum similarity as the first target response text.

[0123] In another embodiment, based on the similarity of multiple first response texts, the first target response text can be obtained by first extracting the image feature P1 of the multimodal data to be annotated and the text feature A of each first response text. n Then, based on the image features of the multimodal data to be annotated and the text features of any first response text, the similarity S between the first response text and the multimodal data to be annotated is calculated. n ; Finally, the first response text corresponding to the maximum similarity among the calculated similarities is used as the first target response text.

[0124] Specifically, Figure 8 As shown, it is a schematic diagram of another process of determining a first target response text provided in an embodiment of the present application;

[0125] S801: Determine whether the number of first response texts is multiple, if so, execute S802, otherwise end;

[0126] S802: extracting image features P1 of the multimodal data to be annotated and text features An of each first response text;

[0127] S803: Calculate the similarity between P1 and the text feature An of each first response text;

[0128] S804: Select the maximum similarity;

[0129] S805: Taking the first response text corresponding to the maximum similarity as the first target response text.

[0130] For example, the multimodal data to be annotated is a picture of a market scene with multiple stalls and crowds; there are three first response texts, namely:

[0131] First response text T1: "The market is bustling with people, and various stalls are filled with fresh fruits and vegetables."

[0132] First response text T2: "A busy market with vendors hawking their wares."

[0133] First response text T3: "The market is bustling with customers selecting various goods."

[0134] The image feature P1 of the multimodal data to be annotated and the text features of the three first response texts are extracted, namely text feature A1, text feature A2 and text feature A3; the similarity calculation results are as follows:

[0135] The first similarity S1 between the multimodal data to be annotated and T1 is 0.89;

[0136] The second similarity S2 between the multimodal data to be annotated and T2 is 0.85;

[0137] The third similarity S3 between the multimodal data to be annotated and T3 is 0.91;

[0138] After comparison, S3 has the maximum similarity, and the first response text T3 is used as the target response text.

[0139] like Fig. 9 As shown, it is a schematic diagram of another process of determining the first target response text provided by the above embodiment, which is as follows:

[0140] S901: Generate three first response texts T1, T2, T3;

[0141] S902: Extracting image features P1 of the multimodal data to be annotated;

[0142] S903: extracting text features A1, A2, A3 of three first response texts;

[0143] S904: Calculate the first similarity S1, the second similarity S2, and the third similarity S3 of the three first response texts;

[0144] S905: Select the maximum similarity from S1, S2, and S3;

[0145] S906: Use the response text corresponding to the maximum similarity as the first target response text.

[0146] After obtaining the first target response text, the response text is displayed on the display interface. In a specific implementation, the response text can be used as the final response text. The first response text can also be edited and the edited response text can be used as the final response text.

[0147] Specifically, in response to the user's third editing operation, the first target response text is edited to obtain the second target response text; and the second target response text is displayed on the display interface.

[0148] After obtaining the second target response text, the multimodal data to be annotated, the first instruction text and the second target response text are used as a sample in the multimodal instruction data set.

[0149] For example, after the display interface displays the first target response text, the user can edit the incoherent parts of the first target response text, use the edited first target response text as the second target response text, and display the second response text on the final display interface, using the multimodal data to be annotated, the first instruction text, and the second target response text as a sample of the multimodal instruction data, such as Fig.10 , which is a schematic diagram of a sample structure of multimodal instruction data provided in an embodiment of the present application.

[0150] Optionally, in one embodiment, the first target response text is obtained and displayed on a display interface, and the multimodal data to be annotated, the first instruction text and the first target response text are used as a sample in the multimodal instruction data set, such as Fig.11 , which is a schematic diagram of a sample structure of another multimodal instruction data provided in an embodiment of the present application.

[0151] In one embodiment, after the first target response text is displayed on the display interface, a marked area can be displayed on the display interface in response to a user's marking operation on the multimedia resource; a third instruction text is generated in response to a fourth editing operation of the user, and the third instruction text is displayed on the display interface; based on the multimodal data corresponding to the marked area and the third instruction text, at least one multimodal algorithm is used to generate at least one second response text;

[0152] In a specific implementation, if a second response text is generated, the second response text is used as the third target response text; if multiple second response texts are generated, the third target response text is obtained based on the similarity of the multiple second response texts or by fusing the multiple second response texts; the third target response text is displayed on the display interface.

[0153] Specifically, after generating the first target response text, the user can mark the multimedia resource, display the marked area on the display interface, and then enter the third instruction text on the display interface, and generate at least one second response text based on the multimodal data and the third instruction text using at least one multimodal algorithm. If a second response text is generated, the third target response text can be output after editing; if multiple second response texts are generated, the third target response text can be obtained based on the similarity of multiple second response texts or by fusing multiple second response texts using the above method.

[0154] It should be noted that if multiple second response texts are generated, the method for obtaining the third target response text based on the similarity of the multiple second response texts or by fusing the multiple second response texts can refer to the above-mentioned method of "if multiple first response texts are generated, the first target response text is obtained based on the similarity of the multiple first response texts or by fusing the multiple first response texts", which will not be repeated here.

[0155] Optionally, the multimodal data to be annotated, the first instruction text, the third instruction text, the first target response text and the third target response text are taken as a sample in the multimodal instruction data set, such as Fig.12 , which is a schematic diagram of a sample structure of another multimodal instruction data provided in an embodiment of the present application.

[0156] For example, Fig.13 As shown, after the display interface displays the first target response text, if the user wants to explain the status of the traffic light displayed by the multimedia resource, he can switch to the editing mode, and then use the rectangular tool on the display interface to frame the traffic light in the marked area of ​​the display interface, and give the third instruction text "Please describe the status of the traffic light within the coordinates" in the instruction text area, and click Submit. The background receives the rectangular coordinates of the front end and edits the third instruction text. The edited third instruction text is "Please describe the status of the traffic light within the coordinate range [517, 30, 619, 89]", and the model generates the second response text based on this: "The current traffic light is green. This means that vehicles can continue to move forward, and pedestrians and other traffic participants need to abide by traffic rules to ensure safe passage. A green traffic light usually means that there is no obstruction to traffic and vehicles can drive normally."

[0157] In a possible embodiment, after the third target response text is displayed on the display interface, the third target response text can be edited in response to the user's fifth editing operation to obtain a fourth target response text; and the fourth target response text is displayed on the display interface.

[0158] Specifically, after the display interface displays the third target response text, the user can edit the erroneous parts of the third target response text, and use the edited third target response text as the fourth target response text and display it on the display interface.

[0159] Optionally, the multimodal data to be annotated, the first instruction text, the third instruction text, the first target response text and the fourth target response text are taken as a sample in the multimodal instruction data set, such as Fig.14 , which is a schematic diagram of a sample structure of another multimodal instruction data provided in an embodiment of the present application.

[0160] For example, Fig.15As shown, use the rectangle tool to frame the sign in the image, and give the third instruction text "Please identify the text in the area" in the instruction text area, click the Submit button, and the background model returns the third target response text according to the third instruction text: electric vehicles, quasi-three-wheeled motorcycles, internal combustion sightseeing vehicles 7:00-22:00, and then some text recognition errors are made in the third target response text. The user can click the Edit button to modify the third target response text and use it as the fourth target response text. The fourth target response text is "electric vehicles, taxis, quasi-three-wheeled motorcycles, internal combustion sightseeing vehicles 7:00-22:00".

[0161] Among them, tool selection and common operations in annotation can be based on shortcut keys, which can control tool selection and common operation functions, such as Fig.16 The figure shows a schematic diagram of a shortcut menu of a display interface provided in an embodiment of the present application.

[0162] In one embodiment, for example, when the multimodal data to be annotated is a video-text multimodal, the user can check the accuracy of the response text frame by frame. If an error occurs in a certain frame, the video can be paused and the response text of the current frame can be modified.

[0163] If you need to further describe a target in the video, you can select the rectangle or polygon tool on the display interface to select the area of ​​interest, such as Fig.17 As shown, after specifying the area with the polygon tool, enter the third instruction text: Please describe the vehicle information in the area in detail, including color, type, license plate number, etc., and then generate the fourth target response text: There are 2 vehicles in the area, parked closely together, one is a white car with license plate number 123456, and the other is also a white car with license plate number 654321.

[0164] Specifically, the control of video playback can be based on shortcut keys, such as Fig.18 As shown, this is a schematic diagram of another display interface shortcut key menu provided in an embodiment of the present application. The space bar can be used to control video playback and pause, and can control forwarding or rewinding one frame, five frames, or ten frames, so as to facilitate accurate positioning of the video at a specified position.

[0165] Based on the same concept, an embodiment of the present application provides an electronic device. The principle of the electronic device in solving technical problems is the same as the principle of the above-mentioned multimodal annotation method in solving technical problems. The implementation of the electronic device can refer to the implementation of the multimodal annotation method, and the repeated parts will not be repeated.

[0166] like Fig.19 FIG. 1 is a schematic diagram of a structure of an electronic device provided in an embodiment of the present application, and the electronic device includes:

[0167] A memory 191, used for storing computer programs;

[0168] The processor 192 is configured to call the computer program stored in the memory and execute according to the obtained program:

[0169] In response to the user's automatic annotation operation, at least one first response text is generated based on the first instruction text and the multimodal data to be annotated, using at least one multimodal algorithm; if one first response text is generated, the first response text is used as the first target response text; if multiple first response texts are generated, the first target response text is obtained based on the similarity of the multiple first response texts or by fusing the multiple first response texts;

[0170] The display 193 is used to respond to the user's operation, display the first instruction text and the multimedia resources corresponding to the multimodal data to be annotated on the display interface; and display the first target response text on the display interface.

[0171] The present application provides a multimodal data annotation method and an electronic device. In response to a user's operation, a first instruction text and a multimedia resource corresponding to the multimodal data to be annotated are displayed on a display interface, which may be a picture or a video. Then, in response to the user's automatic annotation operation, at least one first response text is generated based on the first instruction text and the multimodal data to be annotated using at least one multimodal algorithm. If a first response text is generated, the first response text is used as a first target response text. If multiple first response texts are generated, the first target response text is obtained based on the similarity of the multiple first response texts or by fusing the multiple first response texts, and the first target response text is displayed on the display interface. Therefore, the problem of time-consuming and low efficiency of manual annotation in multimodal data annotation is solved, and the reliability and wide application of multimodal data annotation are improved.

[0172] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods or computer program products. Therefore, the present application may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present application may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.

[0173] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0174] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0175] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0176] Although the preferred embodiments of the present application have been described, those skilled in the art may make other changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications falling within the scope of the present application.

[0177] Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is also intended to include these modifications and variations.

Claims

1. A multimodal data annotation method, characterized in that: include: In response to the user's operation, displaying the first instruction text and the multimedia resources corresponding to the multimodal data to be annotated on the display interface; In response to the user's automatic annotation operation, based on the first instruction text and the multimodal data to be annotated, at least three multimodal algorithms are used to generate at least three first response texts; Based on the similarities of the at least three first response texts, obtaining a first target response text; Displaying the first target response text on the display interface; The obtaining of the first target response text based on the similarity of the at least three first response texts includes: Extracting text features of each first response text; For any two first response texts, based on the text features of the two first response texts, calculating the similarity of the two first response texts; Any one of the two first response texts with the greatest similarity is used as the first target response text, or the two first response texts with the greatest similarity are merged to obtain the first target response text.

2. The method according to claim 1, characterized in that The method further includes: Displaying a second instruction text in a preset template on the display interface; The step of responding to the user's operation and displaying the first instruction text on the display interface includes: In response to a user's selection operation, using the second instruction text as the first instruction text; or In response to a first editing operation by the user, the second instruction text is edited to obtain the first instruction text; or In response to the user's second editing operation, the first instruction text is generated.

3. The method according to claim 1, characterized in that After the first target response text is displayed on the display interface, the method further includes: In response to a third editing operation of the user, the first target response text is edited to obtain a second target response text; The displaying the first target response text on the display interface includes: Displaying the second target response text on the display interface; The method further comprises: The multimodal data to be annotated, the first instruction text and the second target response text are taken as a sample in a multimodal instruction data set.

4. The method according to claim 1, characterized in that: After the display interface displays the first target response text, the method further includes: In response to a user's marking operation on the multimedia resource, displaying a marked area on the display interface; In response to a fourth editing operation of the user, generating a third instruction text, and displaying the third instruction text on the display interface; Based on the multimodal data corresponding to the marked area and the third instruction text, at least one multimodal algorithm is used to generate at least one second response text; If one second response text is generated, the second response text is used as the third target response text; if multiple second response texts are generated, the third target response text is obtained based on the similarity of the multiple second response texts or by fusing the multiple second response texts; The third target response text is displayed on the display interface.

5. The method according to claim 4, characterized in that After the third target response text is displayed on the display interface, the method further includes: In response to a fifth editing operation of the user, the third target response text is edited to obtain a fourth target response text; The fourth target response text is displayed on the display interface.

6. The method according to claim 1, characterized in that The method further includes: The display interface shows some or all of the following: Application scenarios, data types, template types, model settings, and operation instructions.

7. An electronic device, characterized in that: include: Memory for storing computer programs; The processor is used to call the computer program stored in the memory and execute according to the obtained program: In response to the user's automatic annotation operation, at least three first response texts are generated based on the first instruction text and the multimodal data to be annotated by using at least three multimodal algorithms; based on the similarity of the at least three first response texts, a first target response text is obtained; A display, configured to respond to a user's operation, display the first instruction text and the multimedia resources corresponding to the multimodal data to be annotated on a display interface; and display the first target response text on the display interface; The processor is specifically used for: Extracting text features of each first response text; For any two first response texts, based on the text features of the two first response texts, calculating the similarity of the two first response texts; Any one of the two first response texts with the greatest similarity is used as the first target response text, or the two first response texts with the greatest similarity are merged to obtain the first target response text.

Images