A text annotation method and device

Abstract

The present application provides a text annotation method and device, which relates to the field of artificial intelligence and is used to improve the efficiency of data annotation. In this method, a processing device can send original data to multiple annotation devices respectively, and receive first annotation results of the original data from multiple annotation devices. The processing device can also send the annotation results with conflicting annotation contents and / or the annotation results with empty annotation contents in at least two first annotation results to a display device for manual annotation. The processing device can also receive a second annotation result that is manually annotated, and integrate the first annotation result and the second annotation result to obtain a third annotation result. Based on this scheme, the same original data can be annotated by multiple annotation devices, and all the original data does not need to be manually annotated by the annotator. The annotator can only correct a small number of annotation results with conflicting annotation contents and empty annotation contents after annotation by the annotation device, which can improve the efficiency of data annotation.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence, and in particular to a text annotation method and device. Background Art

[0002] At present, smart SMS is widely used in people's daily life, such as hotel reservation, flight inquiry, logistics inquiry and other services. Smart SMS can reduce user operations and provide greater convenience for users. Smart SMS recognition can recognize the content of SMS through the trained recognition engine, and extract and reassemble the keywords of the SMS content.

[0003] However, in order to train the recognition engine, it is usually necessary to train a large number of labeled data sets based on artificial intelligence (AI) systems. These labeled data sets are manually labeled, and the workload of manual labeling is often very large and requires a lot of manpower. Summary of the invention

[0004] The embodiments of the present application provide a text annotation method and device to improve the efficiency of data annotation.

[0005] In the first aspect, an embodiment of the present application provides a text annotation method, which can be executed by a processing device in an embodiment of the present application. In the method, the processing device can send the original data to at least two annotation devices respectively. The processing device can receive the first annotation result obtained by annotating the original data by sending the at least two annotations respectively. The processing device can also send the annotation results with conflicting annotation contents and / or the annotation results with empty annotation contents in at least two of the first annotation results to a display device for display, so that the annotation personnel can manually correct the annotation results with conflicting annotation contents and / or the annotation results with empty annotation contents through the display device, thereby obtaining a second annotation result. The annotation content here can refer to the annotation content of the annotation items annotated by the at least two annotation devices. The processing device can also receive the second annotation result sent by the display device, and integrate the first annotation result and the second annotation result to obtain a third annotation result. The second annotation result here is obtained by the display device based on the user correcting the displayed annotation results with conflicting annotation contents and / or the annotation results with empty annotation contents.

[0006] Based on this solution, the same original data can be synchronously annotated by multiple annotation devices, and among the annotation results obtained by the annotation devices for annotating the original data, the annotation results with conflicting annotation contents and those with empty annotation contents are sent to the display device, so that the annotators can manually correct the annotation results with conflicting annotation contents and / or those with empty annotation contents through the display device. Therefore, it is not necessary for the annotators to manually annotate all the original data, and the annotators can manually correct only a small number of annotation results with conflicting annotation contents and those with empty annotation contents, thereby reducing the workload of manual annotation and improving the efficiency of data annotation.

[0007] In a possible implementation, the processing device may also send the third annotation result to an AI system, and the AI ​​system may train the third annotation result to obtain a recognition engine. The recognition engine here may be used by the intelligent SMS system to recognize SMS content.

[0008] Based on this solution, the AI ​​system can be used to train the annotation results of the original data to obtain a recognition engine. Since the original data does not need to be manually annotated, the amount of original data can be increased to increase the accuracy of the trained recognition engine.

[0009] In a possible implementation manner, if the format of the original data is a format that is unavailable to the at least two annotation devices, the processing device may further convert the format of the original data into a format available to the at least two annotation devices.

[0010] Based on this solution, the processing device can convert the format of the original data so that the labeling device can label the original data to obtain the labeling results.

[0011] In a possible implementation, the processing device may further cyclically execute the following operations until a preset loop iteration condition is met:

[0012] The processing device may send the second annotation result and the original data to the at least two annotation devices, so that the at least two annotation devices re-annotate the original data according to the second annotation result to obtain a fourth annotation result. The processing device may receive the fourth annotation results sent by the at least two annotation devices respectively, and send the annotation results with conflicting annotation contents and / or the annotation results with empty annotation contents in at least two of the fourth annotation results to the display device for display, so that the annotator can manually correct the annotation results with conflicting annotation contents and / or the annotation results with empty annotation contents through the display device to obtain a fifth annotation result. The processing device may also receive the fifth annotation result sent by the display device, and the fifth annotation result is obtained by the display device based on the user correcting the displayed annotation results with conflicting annotation contents and / or the annotation results with empty annotation contents. The processing device may use the fifth annotation result as the new second annotation result, and return to execute the operation of sending the new second annotation result and the original data to the at least two annotation devices.

[0013] Accordingly, when the processing device integrates the first annotation result and the second annotation result to obtain the third annotation result, it may integrate the fourth annotation result obtained in the last iteration and the fifth annotation obtained in the last iteration to obtain the third annotation result.

[0014] Based on this solution, the original data can be cyclically labeled through the above-mentioned cyclic iterative operation, thereby improving the accuracy of the labeling results, obtaining more accurate labeling results and more accurately training the recognition engine.

[0015] In a possible implementation, the processing device may increase the initial number of iterations by 1 each time the operation is executed in a loop. The preset loop iteration condition includes that the number of iterations reaches a first threshold. Alternatively, if the number of original data is greater than or equal to 2, the preset loop iteration condition includes that after executing the operation once, the number of the fifth annotation results obtained is less than or equal to a second threshold.

[0016] For example, if there are 10 original data, the processing device can send the 10 original data to the annotation device for annotation to obtain a first annotation result. For each original data, if at least two first annotation results of the original data include annotation results with conflicting annotation contents and / or annotation results with empty annotation contents, the processing device can send the annotation results with conflicting annotation contents and / or annotation results with empty annotation contents of the original data to the display device for display, and obtain a second annotation result. The processing device can send the second annotation result and the 10 original data to the annotation device for annotation to obtain a fourth annotation result. If the fourth annotation result includes annotation results with conflicting annotation contents and / or annotation results with empty annotation contents, the processing device can send the annotation results with conflicting annotation contents and / or annotation results with empty annotation contents of the original data to the display device for display, and obtain a fifth annotation result. If the number of the fifth annotation results obtained by the processing device is less than the second threshold, the loop operation can be terminated.

[0017] Based on this solution, the loop operation can be terminated when the above-mentioned preset loop iteration conditions are met, and the labeling results of the labeling device and the manual labeling obtained in the last iteration can be integrated to obtain more accurate labeling results of the original data.

[0018] In one possible implementation, if the number of original data is greater than or equal to 2, then when the processing device sends the second annotation result and the original data to the at least two annotation devices, it can send the second annotation result and the original data corresponding to the second annotation result to the at least two annotation devices, or it can also send the second annotation result and all annotation devices to at least two annotation devices.

[0019] Based on this solution, the processing device can send the manually labeled results to the labeling device, so that the labeling device can modify the labeling principle according to the manually labeled results, which can improve the labeling accuracy of the labeling device. In addition, when executing the loop operation, all the original data can be sent to the labeling device or only the manually labeled original data can be sent to the labeling device to reduce the amount of calculation.

[0020] In a possible implementation, when the processing device sends the original data to at least two labeling devices, the processing device may send the original data to the at least two labeling devices through interfaces corresponding to the at least two labeling devices; wherein each labeling device corresponds to one interface.

[0021] Based on this solution, since the communication protocols of the annotation devices may be different, the processing device may provide multiple interfaces and communicate with the annotation devices with different communication protocols through different interfaces.

[0022] In a second aspect, an embodiment of the present application further provides a text annotation device, which can be used to perform the operations in the above-mentioned first aspect and any possible implementation of the first aspect. For example, the device may include modules or units for performing each operation in the above-mentioned first aspect or any possible implementation of the first aspect. For example, it includes a processing unit and a communication unit.

[0023] In a third aspect, an embodiment of the present application further provides a text annotation system, comprising the text annotation apparatus of the second aspect described above and at least two annotation devices, as well as a display device, etc.

[0024] In a fourth aspect, an embodiment of the present application provides a chip system, comprising a processor and optionally a memory; wherein the memory is used to store computer programs, and the processor is used to call and run computer programs from the memory, so that a device equipped with the chip system can execute any method of the above-mentioned first aspect or any possible implementation of the first aspect.

[0025] In the fifth aspect, an embodiment of the present application provides a computer program product, which includes: computer program code, when the computer program code is executed by the communication unit, processing unit or transceiver, or processor of the device, the device can execute any method in the above-mentioned first aspect or any possible implementation of the first aspect.

[0026] The beneficial effects of the above-mentioned second to fifth aspects and their possible designs can be referred to the above-mentioned description of the corresponding beneficial effects of the first aspect and any possible design thereof, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Figure 1 A schematic diagram of an interface for text annotation in the prior art;

[0028] Figure 2 One of the schematic diagrams of the text annotation system provided in the embodiment of the present application;

[0029] Figure 3 One of the exemplary flow charts of the text annotation method provided in the embodiment of the present application;

[0030] Figure 4 A schematic diagram of a display interface of a text annotation method provided in an embodiment of the present application;

[0031] Figure 5 One of the exemplary flow charts of the text annotation method provided in the embodiment of the present application;

[0032] Figure 6 One of the schematic diagrams of the text annotation system provided in the embodiment of the present application;

[0033] Figure 7 One of the exemplary flow charts of the text annotation method provided in the embodiment of the present application;

[0034] Figure 8 One of the schematic diagrams of the text annotation device provided in the embodiment of the present application;

[0035] Fig. 9 This is one of the schematic diagrams of the text annotation device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0036] In order to make the purpose, technical solutions and advantages of the present application clearer, the technical solutions in the embodiments of the present application will be described in detail below in conjunction with the drawings in the embodiments of the present application.

[0037] Before introducing the embodiments of the present application, some terms in the present application are first explained to facilitate understanding by those skilled in the art.

[0038] 1) Labeling, also known as data labeling, refers to the classification, framing, annotation, etc. of data. It usually includes classification labeling, framing labeling, area labeling, point labeling and other labeling.

[0039] The term “plurality” in the embodiments of the present application may refer to two or more than two.

[0040] Reference Figure 1 As shown in FIG. 1 , it is a schematic diagram of the annotation interface for data annotation by a manual annotation platform in the prior art. Figure 1 As shown in a in the figure, it is the original data set. The original data set includes one or more text messages. First, you can select the text messages that need to be annotated, and the manual annotation platform will segment the content of the text messages. For example, the manual annotation platform can segment the content of the text messages through semantic recognition technology and segment the content of the text messages into different words. Figure 1 The option b in the figure is the one to be manually marked. The marking personnel can mark the cut text messages in the option to be manually marked. For example, Figure 1 The SMS selected in a is the SMS of successful hotel reservation. Figure 1In b, the first-level classification of the text message can be marked as hotel, and the second-level classification can be marked as successful hotel booking. The labeling personnel need to complete the labeling of all the items to be labeled. In this way, after the original data set is labeled by the manual labeling platform, the labeled data is trained based on the AI ​​system. However, the number of labeling items for a text message is large. For example, for the notification text message of successful movie ticket booking, the labeling items may include the first-level classification "movie ticket", the second-level classification "movie ticket booking successful", and the labeling value "order number, ticket collection code, verification code, mobile phone number, ticket collection voucher, viewing time, movie name, cinema, hard, seat number, ticket collection place, cinema address", etc.

[0041] Therefore, due to the large number of annotation items and the complex content, manual annotation will waste a lot of manpower and the annotation efficiency is also low. In view of this, the present application embodiment provides a text annotation method and device. Figure 2 Take the example to describe in detail the labeling system applicable to the embodiment of the present application. Figure 2 As shown, the annotation system 200 may include a processing device 201, an annotation device 202, an annotation device 204, and a display device 203. Optionally, the annotation system may further include an annotation device 206 and a display device 205.

[0042] In the annotation system 200, the processing device 201 can receive multiple raw data, and send the received multiple raw data to the annotation device 202, the annotation device 204, and the annotation device 206. For example, if the processing device 201 receives 100,000 raw data, the processing device 201 can send the 100,000 raw data to the annotation device 202, the annotation device 204, and the annotation device 206 respectively.

[0043] The labeling device 202, the labeling device 204, and the labeling device 206 can respectively label multiple original data to obtain labeling results. The labeling device 202, the labeling device 204, and the labeling device 206 can also send the obtained labeling results to the processing device 201. The processing device 201 can summarize the labeling results sent by the above-mentioned labeling devices 202, the labeling devices 204, and the labeling devices 206.

[0044] The processing device 201 may send the annotation results of the annotation items with empty annotation contents to the display device 203 or the display device 205, and may also send the annotation results of the annotation contents with conflicting annotation contents to the display device 203 or the display device 205. For example, the annotation contents of the annotation item "order number" in the annotation results of a certain original data by the annotation device 202 are 10877768723, while the annotation contents of the annotation item "order number" in the annotation results of the original data by the annotation device 204 are A software. Then, the processing device 201 may determine that there are conflicting annotation contents in the annotation results for the original data, and therefore, the processing device 201 may respectively aggregate the annotation results of multiple annotation devices (such as the annotation device 202, the annotation device 204, and the annotation device 206), and send the annotation items with empty annotation contents or conflicting annotation contents to the display device 203 or the display device 205.

[0045] The display device 203 or the display device 205 can display the received annotation items, and the annotator can manually annotate the annotation contents with conflicting annotation contents on the display device 203 or the display device 205 to avoid conflicts; and manually add annotations to the annotation items with empty annotation contents. The annotator can send the manually annotated contents to the processing device 201 through the display device 203 or the display device 205. The processing device 201 can integrate the manually annotated contents and the results obtained by summarizing the annotation results of the annotation devices 202, 204, and 206 to obtain the final annotation results of annotating multiple raw data. The processing device 201 can send the final annotation results to the intelligent SMS server, and the intelligent SMS server trains the annotation results based on the AI ​​system to obtain a recognition engine.

[0046] The labeling device (202, 204, 206) may be a device that labels the collected data set by classification, drawing frames, annotations, etc. It should be understood that Figure 2 The simplified schematic diagram is only used for the sake of understanding. The marking system may further include more marking devices or more display devices. Figure 2 Not drawn in.

[0047] It should be understood that the text annotation method provided in the embodiments of the present application can be applied to annotating text, and can also be used to annotate other data such as voice and pictures. For example, when annotating pictures using the technical solution provided in the embodiments of the present application, Figure 2The annotation devices (202, 204, 206) shown in the figure can annotate the pictures. In addition, the technical solution of the present application can also be applied to various semantic recognition systems, such as: intelligent text message recognition system, voice recognition system, etc. Among them, the semantic recognition system may include at least one display device, at least one processing device and at least two annotation devices.

[0048] The present application will present various aspects, embodiments or features around a system that may include multiple devices, components, modules, etc. It should be understood and appreciated that each system may include additional devices, components, modules, etc., and / or may not include all devices, components, modules, etc. discussed in conjunction with the figures. In addition, combinations of these schemes may also be used.

[0049] The network architecture and business scenarios described in the embodiments of the present application are intended to more clearly illustrate the technical solutions of the embodiments of the present application, and do not constitute a limitation on the technical solutions provided in the embodiments of the present application. A person of ordinary skill in the art can appreciate that with the evolution of the network architecture and the emergence of new business scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0050] Figure 3 is an exemplary flow chart of a text annotation method provided in an embodiment of the present application, which may include the following steps:

[0051] Step 301: A processing device receives a plurality of original data.

[0052] The multiple original data here can be sent from the same device for providing original data, or from different devices for providing original data. For example, the processing device can receive 5 original data sent by device A, or it can receive 3 original data sent by device A and 2 original data sent by device B. The formats of the multiple original data can be the same or different. The processing device can convert the formats of the multiple original data into a format that it can process. For example, the processing device can determine the format of the original data. If the format of the original data is not available, the processing device can convert the format of the original data into a format that it can use.

[0053] For example, the format of the data available to the processing device is a text document (txt) format, the format of the original data transmitted by device A is a txt format, and the format of the original data transmitted by device B is a document (DOC) format. The processing device may determine that the format of the original data transmitted by device B is not available, and the processing device may convert the format of the original data transmitted by device B into a txt format.

[0054] In an embodiment of the present application, the processing device may convert the format of the original data in the following manner. For example, the processing device may pre-store a correspondence between a format conversion identifier and a dynamic code. Among them, the format conversion identifier may include the format before conversion, for example, the format conversion identifier may be format A. Alternatively, the format conversion identifier may also include an initial format name before conversion and a target format name after conversion, for example, the format conversion identifier may be format A converted to format B, etc. After receiving the original data, the processing device may determine the format of the original data. If the format of the original data is a format that is not available to the processing device, the processing device may search for the corresponding dynamic code from the pre-stored correspondence between the format conversion identifier and the dynamic code according to the format name of the original data. The processing device may convert the format of the original data according to the dynamic code found.

[0055] It should be understood that the data format conversion method provided in the embodiment of the present application is only an example, and the embodiment of the present application can also use other format conversion methods to convert the format of data.

[0056] Step 302: The processing device sends the multiple original data to multiple labeling devices.

[0057] In one design, the processing device of an embodiment of the present application provides multiple interfaces, and the processing device can communicate with different annotation devices through the multiple interfaces. The different annotation devices here may refer to annotation devices with different communication protocols. For example, BasicFindr and Appen are two annotation devices with different communication protocols. Therefore, the processing device can provide two interfaces, and the processing device can send multiple raw data to the annotation device using the BasicFindr protocol through interface 1, and synchronously send multiple raw data to the annotation device using the Appen protocol through interface 2.

[0058] Step 303: A plurality of labeling devices respectively label the plurality of original data to obtain labeling results of the plurality of original data, wherein the labeling devices may label the original data according to their own labeling principles.

[0059] For example, the original data is [AA] Your order with the last number 00876705 has started to be delivered. Please receive the goods at the specified time without leaving~ Deliveryman: Zhang San, Phone: 151XXXX7368.

[0060] The labeling result obtained by the labeling device 1 for labeling the above raw data can be as follows:

[0061] Logistics status: Delivery has started

[0062] Order number: 00876705

[0063] Contact: Zhang San

[0064] Contact number: 151XXXX7368

[0065] Tips: Please receive the goods at the designated time and don't leave.

[0066] The labeling result obtained by the labeling device 2 for labeling the above raw data can be as follows:

[0067] Physical status: Delivery has started

[0068] Order number: AA

[0069] Contact: Zhang San

[0070] Contact number: 151XXXX7368

[0071] Step 304: The multiple labeling devices send the labeling results to the processing device respectively.

[0072] The labeling device can send the labeling result to the processing device through the interface provided by the processing device. For example, labeling device 1 can send the labeling result to the labeling device through interface 1, and labeling device 2 can send the labeling result to the processing device through interface 2.

[0073] It should also be noted that the formats of the annotation results may be different due to different annotation devices. Optionally, after receiving the annotation results sent by different annotation devices, if the format of a certain annotation result is not available to itself, the processing device may convert the format of the annotation result into a format available to itself.

[0074] Step 305: The processing device integrates the annotation results respectively sent by the multiple annotation devices, and sends the annotation results that meet the preset conditions to the display device.

[0075] The preset conditions here may include the following two conditions, which are described in detail below:

[0076] Precondition 1: There are conflicting annotation contents in the annotation results.

[0077] The conflicting annotation content here may refer to the different annotation content of the same annotation item for the same original data in the annotation results given by different annotation devices. For example, the annotation content of the annotation item "order number" of the original data A by the above-mentioned annotation device 1 is 00876705, and the annotation content of the annotation item "order number" of the original data A by the annotation device 2 is AA. Then, the processing device may consider that there are conflicting annotation contents in the annotation results of the original data by the annotation device 1 and the annotation device 2. Alternatively, if the annotation content of the annotation item "order number" by the annotation device 1 is 00876705, and the annotation content of the annotation item "order number" by the annotation device 2 is empty, then the processing device may also consider that there are conflicting annotation contents in the annotation results of the original data by the annotation device 1 and the annotation device 2. The processing device may send the annotation results of the annotation device 1 and the annotation device 2 to the display device respectively. In one example, the processing device may only send the conflicting annotation content and the corresponding annotation item to the display device to reduce the transmission occupancy.

[0078] Precondition 2: There are annotation items with empty annotation content in the annotation results.

[0079] The annotation content being empty here may mean that the annotation content of the same annotation item of the same original data by all the annotation devices is empty. For example, the annotation content of the annotation item "movie name" of the original data B by the annotation device 1 is empty, and the annotation content of the annotation item "movie name" of the original data B by the annotation device 2 is also empty, then the processing device may deem that there are annotation items with empty annotation content in the annotation result for the original data B. Therefore, the processing device may send the annotation results of the original data B by multiple annotation devices to the display device. In one example, the processing device may only send the annotation items with empty annotation content to the display device, for example, the processing device only sends the annotation item "movie name" to the display device for display, so as to reduce the transmission occupancy.

[0080] In one embodiment, when the processing device integrates the annotation results sent by the annotation devices, if the annotation results of multiple annotation devices for a certain original data are all the same, then the processing device can consider that the annotation of the original data has been completed, and the processing device can directly store the annotation results.

[0081] For example, the original data: [Software A] You have booked a movie at 17:30 on March 12th, Hall 5, Row 14, Seats 21 and 22, XX Cinema, and you can pick up your ticket with your phone number ending in 1111. Open "Movie Buddy" to customize your own ticket. Official customer service phone number: 4001-111-XXX.

[0082] The labeling device 1 and the labeling device 2 respectively perform labeling on the above raw data, and the following labeling results can be obtained:

[0083] Order status: Ticket booking successful

[0084] Ticket collection voucher: 1111

[0085] Movie viewing time: March 12, 17:30

[0086] Movie name: A movie

[0087] Cinema: XX Cinema

[0088] Seat number: Row 14, seats 21 and 22

[0089] Customer service phone number: 4001-111-XXX

[0090] Since the labeling results of the labeling device 1 and the labeling device 2 for the original data are the same, the processing device can directly store the labeling results for the original data.

[0091] From the above, it can be seen that the processing device can send only the annotation results of the annotation contents with conflicting contents and the annotation results of the annotation contents with empty contents to the display device for display. The annotators can manually annotate the annotation contents with conflicting contents and the annotation items with empty contents according to the display of the display device. For example, the annotation content of the annotation item "Order Number" by annotation device 1 is 00876705, while the annotation content of the annotation item "Order Number" by annotation device 2 is AA. The annotators can determine that 00876705 is the order number, and AA is the name of the application. Therefore, the annotators can manually annotate the annotation item "Order Number" and correct the annotation content to 00876705.

[0092] like Figure 4 As shown, it is a schematic diagram of a possible display interface of the display device in the embodiment of the present application. In the display interface, the annotation results of the annotation device 1 and the annotation device 2 can be displayed respectively, and the annotation result of the annotation device 1 corresponds to "recognition engine 1", and the annotation result of the annotation device 2 corresponds to "recognition engine 2". Optionally, the original data of the annotation result can also be displayed in the display interface. The original data can be sent to the display device by the processing device. The annotation personnel can modify each annotation item in the annotation results of the annotation device 1 and the annotation device 2 respectively. For example, the annotation personnel can modify the annotation result AA of the annotation item "Order Number" in the annotation result of the annotation device 2 to 00876705.

[0093] Step 306: The display device sends the manually labeled results to the processing device.

[0094] The manually annotated content here can be the content that the annotator manually annotates the conflicting annotation content and the annotation items with empty annotation content through the display device. The display device can send the manually annotated content to the processing device in response to the annotated content sent by the annotator. The annotated content here can be, for example, Figure 4 If the annotator clicks the "Save All" button after the annotation is completed, the display device can send the manually annotated content to the processing device.

[0095] Step 307: The processing device integrates the manual annotation results and the annotation results of multiple annotation devices to obtain an integrated annotation result.

[0096] When integrating the annotation results, the processing device may integrate the annotation results of multiple annotation devices for the same original data into one annotation result. It should be understood that the above-mentioned one annotation result may include the same annotation content of multiple annotation devices for the same annotation item, as well as the annotation results of manual annotation.

[0097] For example, the original data is [A Software] Your order with the last number 008766 has started to be delivered. Deliveryman: Zhang San, phone number: 133XXXX8888.

[0098] The labeling results of the labeling device 1 on the above raw data are as follows:

[0099] Logistics status: Delivery has started

[0100] Order number: 008766

[0101] Deliveryman: Zhang San

[0102] Tel: 133XXXX8888

[0103] The labeling results of the labeling device 2 on the above raw data are as follows:

[0104] Logistics status: Delivery has started

[0105] Order number: 008766

[0106] Deliveryman: A software

[0107] Tel: 133XXXX8888

[0108] The result of manual labeling for the conflicting labeling item "deliveryman" is "Zhang San", that is, the result of manual labeling is "deliveryman: Zhang San". Therefore, when the processing device integrates the labeling results of the above raw data, it can integrate the same labeling content of the labeling device with the labeling results of manual labeling. The final labeling result after integration is as follows:

[0109] Logistics status: Delivery has started

[0110] Order number: 008766

[0111] Deliveryman: Zhang San

[0112] Tel: 133XXXX8888

[0113] The following Figure 5 The text annotation method provided in the embodiment of the present application is explained. Figure 5 As shown in FIG. 1 , it is an exemplary flow chart of the text annotation method in an embodiment of the present application, which may include the following steps:

[0114] Step 501: The processing device imports the original data set.

[0115] The original dataset here is the same as above Figure 3 The multiple original data in the illustrated method embodiment may be the same. The processing device may receive the original data from multiple devices.

[0116] Step 502: The processing device converts the format of the original data set into a format that can be used by the annotation device. The description of the format conversion can be found in the above Figure 3 Related description in the method embodiment shown.

[0117] Step 503: The processing device sends the original data set to multiple annotation devices. The processing device can send the original data set to multiple annotation devices through different interfaces, and each annotation device can correspond to one interface. The annotation devices can annotate the original data set based on their own annotation principles.

[0118] Step 504: Multiple labeling devices output labeling results respectively.

[0119] Step 505: The processing device integrates the annotation results of multiple annotation devices, and sends the annotation items with conflicting annotation contents and empty annotation contents to the display page for display.

[0120] For annotation items with conflicting annotation contents and annotation items with empty annotation contents, please refer to the above Figure 3 The relevant descriptions in the method embodiment shown are not repeated here. Among them, the annotation items with conflicting annotation contents and the annotation items with empty annotation contents can be sent to the display device for display.

[0121] Step 505: The annotator performs manual annotation through the display page.

[0122] Step 506: The processing device integrates the manual annotation results and the annotation results of multiple annotation devices, and outputs the final annotation results to other systems.

[0123] The other system here may be an AI system, which may train the final labeling result to obtain a recognition engine, and the recognition engine is used to recognize the content of SMS in the intelligent SMS system.

[0124] In a possible implementation, the embodiment of the present application can also perform iterative annotation on the original data to obtain a more accurate annotation result. Figure 3 After executing step 306, the following may also be performed: the processing device sends the manually annotated content to multiple annotation devices, and resends the multiple original data to the multiple annotation devices respectively.

[0125] The annotation device here can regard the manually annotated content as the modification parameters of the annotation principle, and can annotate the new original data more accurately according to the annotation principle and the modification parameters of the annotation principle when annotating the original data next time. Figure 3 The multiple original data received in step 301 are resent to multiple labeling devices for iterative labeling to obtain more accurate labeling results. The following describes how to perform iterative labeling, which may include method 1 and method 2.

[0126] Mode 1: The processing device may resend the original data corresponding to the annotation results containing conflicting annotation contents and the original data corresponding to the annotation results containing empty annotation contents to multiple annotation devices.

[0127] For example, if the labeling results of labeling device 1 and labeling device 2 for original data 1 are the same, and the labeling results for original data 2 are conflicting labeling results, then the processing device can send the manual labeling results of the labeling results for original data 2 to multiple labeling devices, and resend the original data 2 to multiple labeling devices for iterative labeling training.

[0128] In one design, the processing device may consider that the labeling of the original data 1 has been completed, so the processing device may also directly store the labeling results of the original data 1.

[0129] In one embodiment, when the number of iterative annotations is greater than the first threshold, the processing device ends the iteration and integrates the annotation results of the last iteration to obtain the final annotation results for annotating multiple original data. In one example, the initial number of annotation iterations may be 0, and the processing device may add 1 to the number of iterative annotations before sending multiple original data to multiple annotation devices, and determine whether the number of iterative annotations after adding 1 is less than or equal to the first threshold. If the number of iterative annotations after adding 1 is less than or equal to the first threshold, the processing device may send the original data corresponding to the annotation results containing conflicting annotation content and the original data corresponding to the annotation results containing empty annotation content to multiple annotation devices. If the number of iterative annotations is greater than the first threshold, the processing device may end the iteration and store the annotation results of the last iteration. The processing device may integrate the annotation results of the last iteration and the manually annotated content to obtain the final annotation results for annotating multiple original data, and store the final annotation results. In another example, the initial number of iterative annotations may also be 1, and the processing device may determine whether the number of iterative annotations is less than or equal to the first threshold before sending multiple data to multiple annotation devices. If the number of iterative annotations is less than or equal to the first threshold, the processing device may send the original data corresponding to the annotation results containing conflicting annotation content and the original data corresponding to the annotation results containing empty annotation content to multiple annotation devices, and may increase the number of iterative annotations by 1. If the number of iterative annotations after adding 1 is greater than the first threshold, the processing device may end the iteration and store the annotation results of the last iteration. The processing device may integrate the annotation results of the last iteration and the manually annotated content to obtain the final annotation results of annotating multiple original data, and the final annotation results.

[0130] The processing device can also send the final annotation results to the intelligent SMS server, which trains the annotation results based on the AI ​​system to obtain a recognition engine.

[0131] In another embodiment, the processing device may end the iteration when the number of original data corresponding to the annotation results containing conflicting annotation content and the number of original data corresponding to the annotation results containing empty annotation content is less than or equal to the second threshold. The processing device may integrate the annotation results of the last iteration and the manually annotated content to obtain the final annotation results of the multiple original data, and store the final annotation results. The processing device may also send the final annotation results to the intelligent SMS server, which trains the annotation results based on the AI ​​system to obtain a recognition engine.

[0132] For example, the number of original data is 10w, and the annotation device annotates the 10w original data respectively to obtain a first annotation result. Among the first annotation results of the annotation device for the original data, the first annotation results of 800 original data contain conflicting annotation content and / or annotation results with empty annotation content. The processing device can send the first annotation results of the 800 original data to the display device for manual annotation, and obtain the second annotation results of the 800 original data manually annotated. The processing device sends the second annotation results of the 800 original data manually annotated to the annotation device, and resends the 800 original data to the annotation device, so that the annotation device annotates the 800 original data. At this time, among the annotation results of the 800 original data obtained, the first annotation results of 2 original data contain conflicting annotation content and / or annotation results with empty annotation content, then the processing device can send the first annotation results of the 2 original data to the display device for manual annotation, and obtain the second annotation results of the 2 original data manually annotated. Among them, since 2 is less than the second threshold, the processing device can end the iteration, and can integrate the labeling results of the labeling device in the second iteration and the labeling results of manual labeling to obtain the final labeling results.

[0133] It should be understood that the first threshold and the second threshold mentioned above can be predetermined based on empirical values, and this application does not make any specific limitation.

[0134] Mode 2: The processing device may resend the multiple original data to multiple labeling devices.

[0135] For example, if the annotation results of the original data 1 by the annotation device 1 and the annotation device 2 are the same, and there are conflicting annotation contents or annotation items with empty annotation contents in the annotation results of the original data 2, then the processing device can send the manually annotated contents in the annotation results of the original data 2 to multiple annotation devices, and resend the original data 1 and the original data 2 to multiple annotation devices for iterative annotation to obtain more accurate annotation results.

[0136] In one embodiment, when the number of iterative annotations is greater than the first threshold, the processing device ends the iteration and integrates the annotation results of the last iteration to obtain the final annotation result of annotating multiple original data. In one example, the initial annotation iteration number may be 0, and the processing device may add 1 to the iterative annotation number before sending the multiple original data to the multiple annotation devices, and determine whether the iterative annotation number after adding 1 is less than or equal to the first threshold. If the iterative annotation number after adding 1 is less than or equal to the first threshold, the processing device may send the multiple original data to the multiple annotation devices. If the iterative annotation number is greater than the first threshold, the processing device may end the iteration and integrate the annotation results of the last iteration to obtain the final annotation result of annotating multiple original data. In another example, the initial annotation iteration number may also be 1, and the processing device may determine whether the iterative annotation number is less than or equal to the first threshold before sending the multiple data to the multiple annotation devices. If the iterative annotation number is less than or equal to the first threshold, the processing device may send the multiple original data to the multiple annotation devices, and may add 1 to the iterative annotation number. If the number of iterations after adding 1 is greater than the first threshold, the processing device can end the iteration and integrate the labeling results of the last iteration to obtain the final labeling results of the multiple original data.

[0137] The processing device can also send the final annotation results to the intelligent SMS server, which trains the annotation results based on the AI ​​system to obtain a recognition engine.

[0138] In another embodiment, the processing device may terminate the iteration when the total number of conflicting annotation contents and / or annotation items with empty annotation contents in the annotation results of the multiple annotation devices for all the original data is less than or equal to the third threshold. For example, when the total number of conflicting annotation contents is 0 or the total number of annotation items with empty annotation contents is 0, the processing device may terminate the iteration and integrate the annotation results of the last iteration to obtain the final annotation results of the multiple original data.

[0139] For example, the number of original data is 2w, and the annotation device annotates the 2w original data respectively to obtain a first annotation result. Among the first annotation results of the annotation device for the original data, the first annotation results of 100 original data contain conflicting annotation content and / or annotation results with empty annotation content. The processing device can send the first annotation results of the 100 original data to the display device for manual annotation, and obtain the second annotation results of the 100 original data manually annotated. The processing device sends the second annotation results of the 100 original data manually annotated to the annotation device, and resends the 2w original data to the annotation device, so that the annotation device annotates the 2w original data. At this time, among the annotation results of the 2w original data obtained, the first annotation results of 100 original data contain conflicting annotation content and / or annotation results with empty annotation content, then the processing device can send the first annotation results of the 100 original data to the display device for manual annotation, and obtain the second annotation results of the 100 original data manually annotated. Among them, since 100 is less than the third threshold, the processing device can end the iteration, and can integrate the labeling result of the labeling device in the second iteration and the labeling result of manual labeling to obtain the final labeling result.

[0140] It should be understood that the above-mentioned first threshold and third threshold can be predetermined based on empirical values, and this application does not make any specific limitation.

[0141] The following Figure 6 The method for iteratively labeling the original data in the embodiment of the present application is explained and illustrated. In the embodiment of the present application, a labeling platform can be provided. The labeling platform can include a data import module 601, a single-input multiple-output processing module 602, a labeling device 603, a labeling result merging processing module 604, and a data labeling processing module 605.

[0142] The data import module 601 can be used to import multiple raw data. The data import module 601 here can be set in the processing device in the above embodiment. The data import module 601 can convert the format of the raw data into a format that can be used by the annotation device, and send the raw data set to the annotation device 603 through the single-input multiple-output processing module 602.

[0143] The SIMO processing module 602 provides multiple interfaces and can communicate with different annotation devices through the multiple interfaces. The SIMO processing module 602 can also be set in the processing device in the above embodiment, mainly used for data communication with different annotation devices.

[0144] The annotation platform may include multiple annotation devices 603, and the multiple annotation devices 603 may be as follows: Figure 2The labeling device 202, the labeling device 204 and the labeling device 206 in FIG. The labeling device 603 can label multiple original data to obtain labeling results.

[0145] The annotation result merging processing module 604 can be used to integrate the annotation results output by the annotation devices 603. The specific integration method can be found in the above Figure 3 The annotation result merging processing module 604 can also send the annotation items with conflicting annotation contents and the annotation items with empty annotation contents to the data annotation processing module 605 for display. The annotation result merging processing module 604 can also be set in the processing device in the above embodiment.

[0146] The data annotation processing module 605 can display the annotation items with conflicting annotation contents and the annotation items with empty annotation contents, and can receive the annotation contents manually corrected and annotated for the annotation items with conflicting annotation contents and the annotation items with empty annotation contents. The data annotation processing module 605 can be arranged in the display device in the above embodiment.

[0147] The data result merging processing module 604 can also integrate the results of manual annotation and the annotation results of the annotation device 603, and send the integrated data to the single-input multiple-output processing module 602 again for iterative annotation. The data result merging processing module 604 can also integrate the annotation results of the last iteration after the iteration ends. The annotation of the last iteration here can include the annotation results of the annotation device 603, and can also include the annotation results of manual annotation. The data result merging processing module 604 can send the final annotation results to the AI ​​system in the intelligent SMS server, and the AI ​​system trains the final annotation results to obtain a recognition engine. Among them, the data result merging processing module 604 can be set in the processing device in the above embodiment, and can integrate the annotation results of the annotation device and the results of manual annotation to obtain the final annotation results.

[0148] After introducing the multiple modules on the processing device, the text annotation method in the embodiment of the present application is introduced through specific embodiments. Figure 7 FIG. 1 is an exemplary flow chart of a text annotation method in an embodiment of the present application, which may include the following steps:

[0149] Step 701: The data import module imports the original data. The original data here can refer to the relevant description in the above method embodiment.

[0150] Step 702: The data import module converts the format of the original data into a format that can be used by the annotation device.

[0151] Step 703: The data import module sends the original data after format conversion to the single-input multiple-output processing module.

[0152] Step 704: The SIMO processing module sends the original data to the labeling devices respectively.

[0153] Step 705: The labeling device labels the original data to obtain a labeling result.

[0154] Step 706: The annotation devices send the annotation results to the annotation result merging processing module respectively.

[0155] Step 707: the annotation result merging processing module determines the annotation results with conflicting annotation contents and / or the annotation results with empty annotation contents in the annotation results.

[0156] Step 708: The annotation result merging processing module generates a manual annotation task, and sends the annotation results with conflicting annotation contents and / or the annotation results with empty annotation contents to the data annotation processing module.

[0157] Step 709: The data annotation processing module receives the manually labeled results.

[0158] The data annotation processing module may send the annotation results with conflicting annotation contents and / or the annotation results with empty annotation contents to the display device for display. The manually annotated annotation results herein are the annotation results that the annotator manually annotates the annotation results with conflicting annotation contents and / or the annotation results with empty annotation contents through the display interface of the display device.

[0159] Step 710: The data annotation processing module sends the manually labeled results to the single-input multiple-output processing module.

[0160] Step 711: The single-input multiple-output processing module sends the manually labeled results to the labeling device.

[0161] It should be understood that steps 704 to 711 may be executed cyclically until the condition for the end of the iteration is met, thereby obtaining a relatively accurate final labeling result. The condition for the end of the iteration here can be referred to as Figure 3 The method shown is described in the embodiment.

[0162] After obtaining the final annotation result through multiple iterations, the data annotation processing module can send the final annotation result to other systems. Among them, other systems can be, for example, intelligent SMS servers, AI systems, etc., and the AI ​​system can train the final annotation result to obtain a recognition engine.

[0163] Same as above idea, Figure 8As shown, the present application embodiment further provides a text annotation device 800, which can implement the functions of the processing device in the above method embodiment, and thus can also achieve the beneficial effects possessed by the above method embodiment. The text annotation device 800 includes a communication unit 801 and a processing unit 802.

[0164] In one example, the communication unit 801 is used to send the original data to at least two annotation devices respectively; and receive the first annotation results obtained by annotating the original data and sent by the at least two annotation devices respectively. The processing unit 802 is used to determine that at least two of the first annotation results include annotation results with conflicting annotation contents and / or annotation results with empty annotation contents. The communication unit 801 is also used to send the annotation results with conflicting annotation contents and / or annotation results with empty annotation contents among at least two of the first annotation results to a display device for display. And receive the second annotation result sent by the display device. The processing unit 802 is also used to integrate the first annotation result and the second annotation result to obtain a third annotation result.

[0165] The description of the first annotation result and the second annotation result can be as follows: Figure 3 The relevant description in the method embodiment shown will not be repeated here.

[0166] In one design, the communication unit 801 is also used to send the third annotation result to an artificial intelligence AI system, and the AI ​​system trains the third annotation result to obtain a recognition engine, and the recognition engine is used by the intelligent SMS system to identify SMS content.

[0167] In one design, the processing unit 802 is further configured to convert the format of the original data into a format available to the at least two annotation devices if the format of the original data is a format unavailable to the at least two annotation devices.

[0168] In one design, the processing unit 802 is also used to loop the following operations until a preset loop iteration condition is met: sending the second annotation result and the original data to the at least two annotation devices through the communication unit 801, so that the at least two annotation devices re-annotate the original data according to the second annotation result to obtain a fourth annotation result; receiving the fourth annotation results respectively sent by the at least two annotation devices through the communication unit 801; sending at least two of the fourth annotation results with conflicting annotation contents and / or annotation results with empty annotation contents to a display device for display through the communication unit 801; receiving the fifth annotation result sent by the display device through the communication unit 801; using the fifth annotation result as a new second annotation result, and when integrating the first annotation result and the second annotation result to obtain a third annotation result, specifically used to: integrate the fourth annotation result obtained in the last iteration and the fifth annotation obtained in the last iteration to obtain the third annotation result.

[0169] The description of the fourth and fifth marking results and the preset loop iteration conditions can be found in the above description. Figure 3 The relevant description in the method embodiment shown will not be repeated here.

[0170] In one design, if the number of the original data is greater than or equal to 2; when the communication unit 801 sends the second annotation result and the original data to the at least two annotation devices, it is specifically used to: send the second annotation result and the original data corresponding to the second annotation result to the at least two annotation devices; or, send the second annotation result and the original data greater than or equal to 2 to the at least two annotation devices.

[0171] In one design, when sending the original data to at least two annotation devices, the communication unit 801 is specifically used to: send the original data to the at least two annotation devices through interfaces corresponding to the at least two annotation devices; wherein each annotation device corresponds to one interface.

[0172] For the specific execution process of the communication unit 801 and the processing unit 802, please refer to the records in the above method embodiment. The division of modules / units in the embodiments of the present application is schematic, which is only a logical function division. There may be other division methods in actual implementation. In addition, each functional module in each embodiment of the present application can be integrated into a processor, or it can exist physically separately, or two or more modules can be integrated into one module. The above-mentioned integrated modules can be implemented in the form of hardware or in the form of software functional modules.

[0173] In another example, the text annotation device may be a chip system. In an embodiment of the present application, the chip system may be composed of a chip, or may include a chip and other discrete devices. Exemplarily, the text annotation device includes a processor and an interface, which may be an input / output interface. Among them, the processor performs the functions of the above-mentioned processing unit 802, and the interface performs the functions of the above-mentioned communication unit 801. The edge device may also include a memory, the memory is used to store a program that can be run on the processor, and the processor implements the methods of the above-mentioned embodiments when executing the program.

[0174] Same as above idea, Fig. 9 As shown, the embodiment of the present application also provides a text annotation device 900. The text annotation device 900 includes: a communication interface 901, at least one processor 902, and at least one memory 903. The communication interface 901 is used to communicate with other devices through a transmission medium, so that the text annotation device 900 and other devices communicate. The memory 903 is used to store computer programs. The processor 902 calls the computer program stored in the memory 903, and sends and receives data through the communication interface 901 to implement the method in the above embodiment.

[0175] Exemplarily, the memory 903 is used to store a computer program; the processor 902 calls the computer program stored in the memory 903 to execute the method executed by the processing device in the above embodiment. For example, the communication interface 901 can be used to send the original data to at least two annotation devices respectively; and receive the first annotation results obtained by annotating the original data respectively sent by the at least two annotation devices. The processor 902 can be used to determine that at least two of the first annotation results include annotation results with conflicting annotation contents and / or annotation results with empty annotation contents. The communication interface 901 can also be used to send the annotation results with conflicting annotation contents and / or annotation results with empty annotation contents in at least two of the first annotation results to a display device for display, and receive the second annotation result sent by the display device. The processor 902 can also be used to integrate the first annotation result and the second annotation result to obtain a third annotation result.

[0176] The description of the first annotation result and the second annotation result can be as follows: Figure 3 The relevant description in the method embodiment shown will not be repeated here.

[0177] In one design, the communication interface 901 is also used to send the third annotation result to an artificial intelligence AI system, and the AI ​​system trains the third annotation result to obtain a recognition engine, and the recognition engine is used by the intelligent SMS system to identify SMS content.

[0178] In one design, the processor 902 is further configured to convert the format of the original data into a format available to the at least two annotation devices if the format of the original data is a format unavailable to the at least two annotation devices.

[0179] In one design, the processor 902 is also used to loop the following operations until a preset loop iteration condition is met: sending the second annotation result and the original data to the at least two annotation devices through the communication interface 901, so that the at least two annotation devices re-annotate the original data according to the second annotation result to obtain a fourth annotation result; receiving the fourth annotation results respectively sent by the at least two annotation devices through the communication interface 901; sending at least two of the fourth annotation results with conflicting annotation contents and / or annotation results with empty annotation contents to a display device for display through the communication unit 801; receiving the fifth annotation result sent by the display device through the communication interface 901; using the fifth annotation result as a new second annotation result, and when integrating the first annotation result and the second annotation result to obtain a third annotation result, specifically used to: integrate the fourth annotation result obtained in the last iteration and the fifth annotation obtained in the last iteration to obtain the third annotation result.

[0180] The description of the fourth and fifth marking results and the preset loop iteration conditions can be found in the above description. Figure 3 The relevant description in the method embodiment shown will not be repeated here.

[0181] In one design, if the number of the original data is greater than or equal to 2; when the communication interface 901 sends the second annotation result and the original data to the at least two annotation devices, it is specifically used to: send the second annotation result and the original data corresponding to the second annotation result to the at least two annotation devices; or, send the second annotation result and the original data greater than or equal to 2 to the at least two annotation devices.

[0182] In one design, there may be multiple communication interfaces 901, and each communication interface corresponds to a labeling device.

[0183] In the embodiment of the present application, the communication interface 901 can be a transceiver, a circuit, a bus, a module or other types of communication interfaces. The processor 902 can be a general-purpose processor, a digital signal processor, an application-specific integrated circuit, a field programmable gate array or other programmable logic device, a discrete gate or transistor logic device, a discrete hardware component, and can implement or execute the disclosed methods, steps and logic block diagrams in the embodiment of the present application. The general-purpose processor can be a microprocessor or any conventional processor, etc. The steps of the method disclosed in the embodiment of the present application can be directly embodied as a hardware processor to perform, or the hardware and software modules in the processor are combined and performed. The memory 903 can be a non-volatile memory, such as a hard disk drive (HDD) or a solid-state drive (SSD), etc., and can also be a volatile memory (volatile memory), such as a random access memory (RAM). The memory is any other medium that can be used to carry or store the desired program code in the form of an instruction or data structure and can be accessed by a computer, but is not limited thereto. The memory in the embodiment of the present application can also be a circuit or any other device that can realize the storage function. The memory 903 is coupled to the processor 902. The coupling in the embodiment of the present application is an interval coupling or communication connection between devices, units or modules, which can be electrical, mechanical or other forms, and is used for information exchange between devices, units or modules. As another implementation, the memory 903 can also be located outside the device 900. The processor 902 can operate in conjunction with the memory 903. The processor 902 can execute program instructions stored in the memory 903. At least one of the at least one memory 903 may also be included in the processor 902. The connection medium between the above-mentioned communication interface 901, the processor 902 and the memory 903 is not limited in the embodiment of the present application. For example, in the embodiment of the present application, Fig. 9 The memory 903, the processor 902 and the communication interface 901 may be connected via a bus, which may be divided into an address bus, a data bus, a control bus and the like.

[0184] Understandably, the above Figure 8 The text annotation device 800 in the illustrated embodiment can be Figure 7 The text annotation device 900 shown is implemented. Specifically, the processing unit 802 can be implemented by a processor 902 , and the communication unit 801 can be implemented by a communication interface 901 .

[0185] In the method provided in the embodiment of the present application, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present invention is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, a network device, a user device or other programmable device. The computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions may be transmitted from a website site, a computer, a server or a data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, server or data center. The computer-readable storage medium may be any available medium that a computer can access or a data storage device such as a server or a data center that includes one or more available media integrated. The available medium may be a magnetic medium (eg, a floppy disk, a hard disk, a magnetic tape), an optical medium (eg, a digital video disc (DVD)), or a semiconductor medium (eg, a solid state disk SSD), etc.

[0186] As described above, the above embodiments are only used to introduce the technical solution of the present application in detail, but the description of the above embodiments is only used to help understand the method of the embodiments of the present invention and should not be understood as limiting the embodiments of the present invention. Changes or substitutions that can be easily thought of by technicians in this technical field should be included in the protection scope of the embodiments of the present invention.

Claims

1. A text annotation method, characterized in that: include: Sending the original data to at least two annotation devices respectively; Receiving first annotation results respectively sent by the at least two annotation devices and obtained by annotating the original data; Sending at least two of the first annotation results, the annotation results with conflicting annotation contents and / or the annotation results with empty annotation contents, to a display device for display; The annotation content is the annotation content of the annotation item annotated by the at least two annotation devices; receiving a second annotation result sent by the display device, where the second annotation result is obtained by the display device based on correction by a user of an annotation result with conflicting annotation contents and / or an annotation result with empty annotation contents displayed; Integrate the first labeling result and the second labeling result to obtain a third labeling result; wherein, if the labeling results of the at least two labeling devices on the original data are all the same, the third labeling result is the first labeling result obtained by the at least two labeling devices on the original data; if the labeling results of the at least two labeling devices on the original data are not completely the same, the third labeling result includes the first labeling result that is the same among the labeling results of the at least two labeling devices on the original data, and the second labeling result.

2. The method according to claim 1, characterized in that Also includes: The third annotation result is sent to an artificial intelligence AI system, and the AI ​​system trains the third annotation result to obtain a recognition engine, and the recognition engine is used for the intelligent SMS system to recognize SMS content.

3. The method according to claim 1 or 2, characterized in that: Also includes: If the format of the original data is a format that is unavailable to the at least two annotation devices, the format of the original data is converted into a format available to the at least two annotation devices.

4. The method according to claim 1, characterized in that Also includes: The following operations are executed in a loop until the preset loop iteration condition is met: sending the second annotation result and the original data to the at least two annotation devices, so that the at least two annotation devices re-annotate the original data according to the second annotation result to obtain a fourth annotation result; Receiving the fourth annotation results respectively sent by the at least two annotation devices; Sending at least two of the fourth annotation results, the annotation results with conflicting annotation contents and / or the annotation results with empty annotation contents, to a display device for display; receiving a fifth annotation result sent by the display device, where the fifth annotation result is obtained by the display device based on correction by a user of an annotation result with conflicting annotation contents and / or an annotation result with empty annotation contents displayed; Using the fifth annotation result as a new second annotation result; Integrating the first annotation result and the second annotation result to obtain a third annotation result includes: The fourth annotation result obtained in the last iteration and the fifth annotation result obtained in the last iteration are integrated to obtain the third annotation result.

5. The method according to claim 4, characterized in that The operation is performed once per loop, and the initial number of iterations is incremented by 1, and the preset loop iteration condition includes: the number of iterations reaches a first threshold; or The number of the original data is greater than or equal to 2, and the preset loop iteration condition includes: after executing the operation once, the number of the fifth annotation results obtained is less than or equal to a second threshold.

6. The method according to claim 4 or 5, characterized in that: If the number of the original data is greater than or equal to 2; The sending the second annotation result and the original data to the at least two annotation devices includes: Sending the second annotation result and the original data corresponding to the second annotation result to the at least two annotation devices; or The second annotation result and the two or more original data are sent to the at least two annotation devices.

7. The method according to claim 1, 2, 4 or 5, characterized in that: The sending of the original data to at least two annotation devices includes: The original data is sent to the at least two annotation devices through interfaces corresponding to the at least two annotation devices; wherein each annotation device corresponds to one interface.

8. A text annotation device, characterized in that: including a communication unit and a processing unit; The communication unit is used to send the original data to at least two annotation devices respectively; and receive first annotation results obtained by annotating the original data and sent by the at least two annotation devices respectively; The processing unit is configured to determine that at least two of the first annotation results include an annotation result with conflicting annotation contents and / or an annotation result with empty annotation contents; The annotation content is the annotation content of the annotation item annotated by the at least two annotation devices; The communication unit is further used to send at least two of the first annotation results whose annotation contents conflict and / or whose annotation contents are empty to a display device for display; and receive a second annotation result sent by the display device, where the second annotation result is obtained by the display device based on a user correcting the displayed annotation results whose annotation contents conflict and / or whose annotation contents are empty; The processing unit is further used to integrate the first labeling result and the second labeling result to obtain a third labeling result; wherein, if the labeling results of the at least two labeling devices on the original data are all the same, then the third labeling result is the first labeling result obtained by the at least two labeling devices labeling the original data; if the labeling results of the at least two labeling devices on the original data are not completely the same, then the third labeling result includes the same first labeling result among the labeling results of the at least two labeling devices on the original data, and the second labeling result.

9. The device according to claim 8, characterized in that The communication unit is also used for: The third annotation result is sent to an artificial intelligence AI system, and the AI ​​system trains the third annotation result to obtain a recognition engine, and the recognition engine is used for the intelligent SMS system to recognize SMS content.

10. The device according to claim 8 or 9, characterized in that The processing unit is also used for: If the format of the original data is a format that is unavailable to the at least two annotation devices, the format of the original data is converted into a format available to the at least two annotation devices.

11. The device according to claim 8, characterized in that The processing unit is also used for: The following operations are executed in a loop until the preset loop iteration condition is met: Sending the second annotation result and the original data to the at least two annotation devices through the communication unit, so that the at least two annotation devices re-annotate the original data according to the second annotation result to obtain a fourth annotation result; receiving, by the communication unit, the fourth annotation results respectively sent by the at least two annotation devices; Sending, by the communication unit, at least two of the fourth annotation results, the annotation results with conflicting annotation contents and / or the annotation results with empty annotation contents, to a display device for display; receiving, by the communication unit, a fifth annotation result sent by the display device, wherein the fifth annotation result is obtained by the display device based on correction by a user of an annotation result with conflicting annotation contents and / or an annotation result with empty annotation contents displayed; Using the fifth annotation result as a new second annotation result; The processing unit, when integrating the first annotation result and the second annotation result to obtain a third annotation result, is specifically configured to: The fourth annotation result obtained in the last iteration and the fifth annotation obtained in the last iteration are integrated to obtain the third annotation result.

12. The device according to claim 11, characterized in that The processing unit is further configured to execute the operation once per cycle, and increase the initial number of iterations by 1, wherein the preset cycle iteration condition includes: the number of iterations reaches a first threshold; or The number of the original data is greater than or equal to 2, and the preset loop iteration condition includes: after executing the operation once, the number of the fifth annotation results obtained is less than or equal to a second threshold.

13. The device according to claim 11 or 12, characterized in that The number of the original data is greater than or equal to 2; When the communication unit sends the second annotation result and the original data to the at least two annotation devices, the communication unit is specifically configured to: sending the second annotation result and the original data corresponding to the second annotation result to the at least two annotation devices; or The second annotation result and the two or more original data are sent to the at least two annotation devices.

14. The device according to claim 8, 9, 11 or 12, characterized in that When the communication unit sends the original data to at least two annotation devices, it is specifically used to: The original data is sent to the at least two annotation devices through interfaces corresponding to the at least two annotation devices; wherein each annotation device corresponds to one interface.

15. A text annotation device, characterized in that: The device comprises a processor and a memory, wherein the memory stores computer-executable instructions, and when the processor executes the computer-executable instructions, the device executes the method according to any one of claims 1 to 7.

16. A computer storage medium, characterized in that: A computer program or instruction is stored thereon, and when the computer program or instruction is executed by a processor, the processor executes the method according to any one of claims 1 to 7.

Images