A method for extracting text annotations

Abstract

The present application provides a method for text annotation extraction. The method is applied to the field of artificial intelligence. The method includes: the server receives and responds to a first annotation request sent by an electronic device, and starts a dynamic annotation service, where the dynamic annotation service corresponds to a first dynamic annotation model; the server receives a first text sequence sent by the electronic device and first annotation information corresponding to the first text sequence, where the first text sequence includes first text information and a second label; the server obtains a second text sequence; when the number of the first text sequences is greater than a first threshold, the server determines second annotation information of the second text sequence through the first dynamic annotation model; the server extracts text with the attribute of the second label in the second text sequence based on the second annotation information. Through this method, when the annotation labels of the automatic annotation model change, there is no need to retrain the model, which can improve the efficiency and accuracy of automatic annotation.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence (AI), and in particular, to a method for extracting text annotations. Background Art

[0002] At present, with the rapid development of artificial intelligence, in many scenarios, annotators are required to annotate data to obtain a large amount of labeled data, and the model is trained with the labeled data to improve the human-computer interaction experience. Annotating data by annotators (manually tagging data) is an important means to obtain labeled data. By automatically annotating data through model prediction, the annotator only needs to review and modify some data, which can greatly improve the efficiency of annotating data. However, the labels of the data predicted by the model need to train the model with a large amount of labeled data in advance. Obviously, this is contrary to the need for annotators to annotate data to obtain a large amount of labeled data.

[0003] In the case of a small amount of data, it is relatively difficult to extract slots based on traditional machine learning methods for slot extraction, and the label system of the model used for annotating data is fixed during the training of the model, that is, the annotated labels are fixed. If new labels are added or the labels change, the model needs to be retrained.

[0004] Retraining the model takes a certain amount of time, and at the same time, if the training data is newly added, deleted or modified, the model used for automatically annotating data also needs to be adjusted accordingly. Summary of the Invention

[0005] An embodiment of the present invention provides a method for extracting text annotations, which can improve the efficiency and accuracy of automatic annotation without retraining the model when the annotation labels of the automatic annotation model change.

[0006] In a first aspect, an embodiment of the present application provides a method for extracting text annotations. The server receives a first annotation request sent by an electronic device; in response to the first annotation request, the server starts a dynamic annotation service, where the dynamic annotation service corresponds to a first dynamic annotation model, and the first dynamic annotation model is trained by a sample text sequence with a first label; the server receives a first text sequence sent by the electronic device and first annotation information corresponding to the first text sequence, where the first text sequence includes first text information and a second label, and the first annotation information is used to indicate the position of the text with the attribute of the second label in the first text sequence, and the first label is different from the second label; the server obtains a second text sequence; when the number of the first text sequences is greater than a first threshold, the server determines second annotation information of the second text sequence through the first dynamic annotation model based on the first text sequence and the first annotation information, and the second annotation information is used to indicate the position of the text with the attribute of the second label in the second text sequence; based on the second annotation information, the text with the attribute of the second label in the second text sequence is extracted. When the annotation label changes, there is no need to retrain the model, which can improve the efficiency and accuracy of automatic annotation.

[0007] In combination with the first aspect, in a possible implementation manner of the first aspect, the server determines a transition probability between the text with the attribute of the second label and the text without the attribute of the second label in the first text sequence based on the first text sequence and the first annotation information, where the text with the attribute of the second label includes a starting word with the attribute of the second label and a continuing word with the attribute of the second label; the server determines an emission probability between each word in the second text sequence and the text with the attribute of the second label and the text without the attribute of the second label based on the first text sequence and the first annotation information; the server annotates the second annotation information of the second text sequence according to the transition probability and the emission probability.

[0008] Combined with the first aspect, in a possible implementation of the first aspect, the server determines, based on the first text sequence and the first annotation information, the first transfer count between the words without the second label attribute and the words without the second label attribute in the first text sequence, the second transfer count between the words without the second label attribute and the starting word with the second label attribute in the first text sequence, the third transfer count between the words without the second label attribute and the continuing word with the second label attribute in the first text sequence, the fourth transfer count between the starting word with the second label attribute and the words without the second label attribute in the first text sequence, the fifth transfer count between the starting word with the second label attribute and the starting word with the second label attribute in the first text sequence, the sixth transfer count between the starting word with the second label attribute and the continuing word with the second label attribute in the first text sequence, the seventh transfer count between the continuing word with the second label attribute and the words without the second label attribute in the first text sequence, the eighth transfer count between the continuing word with the second label attribute and the starting word with the second label attribute in the first text sequence, and the ninth transfer count between the continuing word with the second label attribute and the continuing word with the second label attribute in the first text sequence; the server smooths the observed values of the first transfer count, the second transfer count, the third transfer count, the fourth transfer count, the fifth transfer count, the sixth transfer count, the seventh transfer count, the eighth transfer count, and the ninth transfer count according to the Laplace smoothing constant.

[0009] The server determines the transfer probability between the words with the second label attribute and the words without the second label attribute in the first text sequence according to the first transfer count after smoothing the observed values, the second transfer count after smoothing the observed values, the third transfer count after smoothing the observed values, the fourth transfer count after smoothing the observed values, the fifth transfer count after smoothing the observed values, the sixth transfer count after smoothing the observed values, the seventh transfer count after smoothing the observed values, the eighth transfer count after smoothing the observed values, and the ninth transfer count after smoothing the observed values.

[0010] In combination with the first aspect, in a possible implementation manner of the first aspect, the server determines the emission probability between each word in the second text sequence and the words with the attribute of the second label and the words without the attribute of the second label based on the first text sequence and the first annotation information. Specifically, it includes: the server obtains the above-context vector according to the first text information; the server obtains the below-context vector according to the second text sequence; the server determines the representation of the label according to the below-context vector and the first annotation information; the server calculates the vector distance between the above-context vector and the representation of the label; the server normalizes the vector distance to determine the emission probability between each word in the second text sequence and the words with the attribute of the second label and the words without the attribute of the second label.

[0011] In combination with the first aspect, in a possible implementation manner of the first aspect, the server annotates the second annotation information of the second text sequence according to the transition probability and the emission probability. Specifically, it includes: the server performs Viterbi solution on the second text sequence according to the Viterbi algorithm to obtain the optimal solution path in the second text sequence; wherein, in the optimal solution path, the emission probability value between each word in the second text sequence and the words with the attribute of the second label and the words without the attribute of the second label is the largest; the server annotates the second annotation information of the second text sequence based on the optimal solution path.

[0012] In combination with the first aspect, in a possible implementation manner of the first aspect, the server obtains a third text sequence;

[0013] The server determines the third annotation information of the third text sequence through the first dynamic annotation model based on the first text sequence, the first annotation information, the second text sequence, and the second annotation information. The third annotation information is used to indicate the position of the words with the attribute of the second label in the third text sequence in the third text sequence; the server extracts the words with the attribute of the second label in the third text sequence based on the third annotation information.

[0014] In combination with the first aspect, in a possible implementation manner of the first aspect, the type of the first annotation information includes the BIO information type.

[0015] In combination with the first aspect, in a possible implementation manner of the first aspect, the type of the first label may include any one or several of time, origin, destination, vehicle type, etc., and the type of the second label may include any one or several of time, origin, destination, vehicle type, etc.

[0016] Second aspect, the present application provides a server, including a processor and a memory; the memory is coupled to the processor, and the memory is used to store computer program code, the computer program code includes computer instructions, and the processor calls the computer instructions to cause the server to perform: receiving a first annotation request sent by an electronic device; in response to the first annotation request, starting a dynamic annotation service, where the dynamic annotation service corresponds to a first dynamic annotation model, and the first dynamic annotation model is trained by a sample text sequence with a first label; receiving a first text sequence sent by the electronic device and first annotation information corresponding to the first text sequence, where the first text sequence includes first text information and a second label, and the first annotation information is used to indicate the position of the text with the attribute of the second label in the first text sequence in the first text sequence, and the first label and the second label are different; obtaining a second text sequence; when the number of the first text sequences is greater than a first threshold, determining, through the first dynamic annotation model, based on the first text sequence and the first annotation information, second annotation information of the second text sequence, where the second annotation information is used to indicate the position of the text with the attribute of the second label in the second text sequence in the second text sequence; extracting, based on the second annotation information, the text with the attribute of the second label in the second text sequence. In this way, when the annotation label of the automatic annotation model changes, there is no need to retrain the model, which can improve the efficiency and accuracy of automatic annotation.

[0017] In combination with the second aspect, in a possible implementation manner of the second aspect, the processor is further configured to call the computer instructions to cause the server to perform: determining, based on the first text sequence and the first annotation information, a transition probability between the text with the attribute of the second label and the text without the attribute of the second label in the first text sequence, where the text with the attribute of the second label includes the starting word with the attribute of the second label and the continuing word with the attribute of the second label; determining, based on the first text sequence and the first annotation information, an emission probability between each word in the second text sequence and the text with the attribute of the second label and the text without the attribute of the second label; and annotating the second annotation information of the second text sequence according to the transition probability and the emission probability.

[0018] In combination with the second aspect, in a possible implementation manner of the second aspect, the processor is further configured to call the computer instructions to cause the server to execute: based on the first text sequence and the first annotation information, determine the first transition count between the words without the second label attribute and the words without the second label attribute in the first text sequence, the second transition count between the words without the second label attribute and the starting word with the second label attribute in the first text sequence, the third transition count between the words without the second label attribute and the continuing word with the second label attribute in the first text sequence, the fourth transition count between the starting word with the second label attribute and the words without the second label attribute in the first text sequence, the fifth transition count between the starting word with the second label attribute and the starting word with the second label attribute in the first text sequence, the sixth transition count between the starting word with the second label attribute and the continuing word with the second label attribute in the first text sequence, the seventh transition count between the continuing word with the second label attribute and the words without the second label attribute in the first text sequence, the eighth transition count between the continuing word with the second label attribute and the starting word with the second label attribute in the first text sequence, and the ninth transition count between the continuing word with the second label attribute and the continuing word with the second label attribute in the first text sequence; smooth the observed values of the first transition count, the second transition count, the third transition count, the fourth transition count, the fifth transition count, the sixth transition count, the seventh transition count, the eighth transition count, and the ninth transition count according to the Laplace smoothing constant.

[0019] Determine the transition probability between the words with the second label attribute and the words without the second label attribute in the first text sequence according to the first transition count after smoothing the observed values, the second transition count after smoothing the observed values, the third transition count after smoothing the observed values, the fourth transition count after smoothing the observed values, the fifth transition count after smoothing the observed values, the sixth transition count after smoothing the observed values, the seventh transition count after smoothing the observed values, the eighth transition count after smoothing the observed values, and the ninth transition count after smoothing the observed values.

[0020] In combination with the second aspect, in a possible implementation of the second aspect, the processor is further configured to call the computer instructions to cause the server to execute: obtaining an upstream vector according to the first text information; obtaining a downstream vector according to the second text sequence; determining a representation of the label according to the downstream vector and the first annotation information; calculating a vector distance between the upstream vector and the representation of the label; normalizing the vector distance to determine an emission probability between each word in the second text sequence and the words with the attribute of the second label and the words without the attribute of the second label.

[0021] In combination with the second aspect, in a possible implementation of the second aspect, the processor is further configured to call the computer instructions to cause the server to execute: performing Viterbi decoding on the second text sequence according to the Viterbi algorithm to obtain an optimal decoding path in the second text sequence; wherein, in the optimal decoding path, the emission probability value between each word in the second text sequence and the words with the attribute of the second label and the words without the attribute of the second label is the largest; annotating the second annotation information of the second text sequence based on the optimal decoding path.

[0022] In combination with the second aspect, in a possible implementation of the second aspect, the processor is further configured to call the computer instructions to cause the server to execute: obtaining a third text sequence; determining the third annotation information of the third text sequence through the first dynamic annotation model based on the first text sequence, the first annotation information, the second text sequence, and the second annotation information, where the third annotation information is used to indicate the position of the words with the attribute of the second label in the third text sequence in the third text sequence; extracting the words with the attribute of the second label in the third text sequence based on the third annotation information.

[0023] In combination with the second aspect, in a possible implementation of the second aspect, the processor is further configured to call the computer instructions to cause the server to execute: the type of the first annotation information includes the BIO information type.

[0024] In combination with the second aspect, in a possible implementation of the second aspect, the processor is further configured to call the computer instructions to cause the server to execute: the type of the first label may include any one or several of time, origin, destination, vehicle type, and the type of the second label may include any one or several of time, origin, destination, vehicle type.

[0025] In a third aspect, a computer-readable storage medium is provided. The computer-readable storage medium stores a computer program. When the computer program is executed by a processor, the processor executes a text annotation extraction method provided in the first aspect and any one of the implementation manners in combination with the first aspect.

[0026] In a fourth aspect, a computer program product is provided. The computer program product includes instructions. When the computer program product is executed by a computer, the computer can execute a text annotation extraction method provided in the first aspect and any one of the implementation manners in combination with the first aspect. Description of the Drawings

[0027] To more clearly illustrate the technical solutions of the embodiments of the present invention, the following will briefly introduce the drawings required for the description of the embodiments. Obviously, the following drawings are some embodiments of the present invention. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.

[0028] Figure 1 It is a schematic structural diagram of an electronic device 100 provided by an embodiment of the present application;

[0029] Figure 2 It is a schematic diagram of the relationship between slots and slot values provided by an embodiment of the present application;

[0030] Figure 3 It is a schematic diagram of a system architecture provided by an embodiment of the present application;

[0031] Figure 4 It is a flowchart of slot labels for annotating unannotated text sequences provided by an embodiment of the present application;

[0032] Figure 5 It is a schematic diagram of the annotation result of a text sequence provided by an embodiment of the present application;

[0033] Figures 6 - 9 It is a schematic diagram of a set of transition matrices provided by an embodiment of the present application;

[0034] Figures 10 - 11 It is a schematic diagram of a set of emission matrices provided by an embodiment of the present application;

[0035] Figure 12 It is a schematic diagram of slot labels for annotating unannotated text sequences provided by an embodiment of the present application;

[0036] Figure 13 It is a flowchart of a text annotation extraction method provided by an embodiment of the present application;

[0037] Figure 14An exemplary user interface that provides a selection entry for the user on the electronic device 100. Detailed implementation manners

[0038] The technical solutions in the embodiments of the present application will be described clearly and in detail below with reference to the accompanying drawings. Among them, in the description of the embodiments of the present application, unless otherwise specified, " / " means "or". For example, A / B may mean A or B; "and / or" in the text is only a description of the association relationship of the associated objects, indicating that there can be three relationships. For example, A and / or B may mean: A exists alone, A and B exist simultaneously, and B exists alone. In addition, in the description of the embodiments of the present application, "a plurality of" means two or more than two.

[0039] Hereinafter, the terms "first" and "second" are only used for descriptive purposes and cannot be construed as indicating or implying relative importance or implicitly specifying the quantity of the indicated technical features. Thus, features defined with "first" and "second" may explicitly or implicitly include one or more of such features. In the description of the embodiments of the present application, unless otherwise specified, the meaning of "a plurality of" is two or more.

[0040] Figure 1 The structural schematic diagram of the electronic device 100 is shown.

[0041] Hereinafter, the embodiments will be specifically described by taking the electronic device 100 as an example. It should be understood that Figure 1 The illustrated electronic device 100 is only an example, and the electronic device 100 may have more or fewer components than Figure 1 those shown, may combine two or more components, or may have different component configurations. The various components shown in the figure may be implemented in hardware, software, or a combination of hardware and software, including one or more signal processing and / or application specific integrated circuits.

[0042] The electronic device 100 may include: a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, a headphone jack 170D, a sensor module 180, a button 190, a motor 191, an indicator 192, a camera 193, a display screen 194, and a subscriber identification module (SIM) card interface 195, etc. Among them, the sensor module 180 may include a pressure sensor 180A, a gyroscope sensor 180B, a barometric pressure sensor 180C, a magnetic sensor 180D, an acceleration sensor 180E, a distance sensor 180F, a proximity light sensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, a touch sensor 180K, an ambient light sensor 180L, a bone conduction sensor 180M, etc.

[0043] It can be understood that the structure schematically shown in the embodiments of the present invention does not constitute a specific limitation on the electronic device 100. In other embodiments of the present application, the electronic device 100 may include more or fewer components than those shown in the figure, or combine certain components, or split certain components, or have different component arrangements. The components shown in the figure may be implemented in hardware, software, or a combination of software and hardware.

[0044] The processor 110 may include one or more processing units. For example, the processor 110 may include an application processor (AP), a modem processor, a graphics processing unit (GPU), an image signal processor (ISP), a controller, a memory, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU), etc. Among them, different processing units may be independent devices or integrated in one or more processors.

[0045] Among them, the controller may be the nerve center and command center of the electronic device 100. The controller may generate operation control signals according to the instruction operation code and timing signal to complete the control of fetching instructions and executing instructions.

[0046] A memory can also be provided in the processor 110 for storing instructions and data. In some embodiments, the memory in the processor 110 is a cache memory. This memory can hold the instructions or data that the processor 110 has just used or recycled. If the processor 110 needs to use the instruction or data again, it can directly call it from the said memory. This avoids repeated accesses, reduces the waiting time of the processor 110, and thus improves the efficiency of the system.

[0047] In some embodiments, the processor 110 may include one or more interfaces. The interfaces may include an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a subscriber identity module (SIM) interface, and / or a universal serial bus (USB) interface, etc.

[0048] The I2C interface is a bidirectional synchronous serial bus, including a serial data line (SDA) and a serial clock line (SCL). In some embodiments, the processor 110 may include multiple groups of I2C buses. The processor 110 can be respectively coupled to the touch sensor 180K, the charger, the flash, the camera 193, etc. through different I2C bus interfaces. For example: the processor 110 can be coupled to the touch sensor 180K through the I2C interface, enabling the processor 110 to communicate with the touch sensor 180K through the I2C bus interface to implement the touch function of the electronic device 100.

[0049] The I2S interface can be used for audio communication. In some embodiments, the processor 110 may include multiple groups of I2S buses. The processor 110 can be coupled to the audio module 170 through the I2S bus to implement communication between the processor 110 and the audio module 170. In some embodiments, the audio module 170 can transmit an audio signal to the wireless communication module 160 through the I2S interface to implement the function of answering a call through a Bluetooth headset.

[0050] The PCM interface can also be used for audio communication to sample, quantize, and encode analog signals. In some embodiments, the audio module 170 and the wireless communication module 160 can be coupled through a PCM bus interface. In some embodiments, the audio module 170 can also transmit audio signals to the wireless communication module 160 through the PCM interface to implement the function of answering a call through a Bluetooth headset. Both the I2S interface and the PCM interface can be used for audio communication.

[0051] The UART interface is a general-purpose serial data bus for asynchronous communication. This bus can be a two-way communication bus. It converts the data to be transmitted between serial communication and parallel communication. In some embodiments, the UART interface is typically used to connect the processor 110 and the wireless communication module 160. For example, the processor 110 communicates with the Bluetooth module in the wireless communication module 160 through the UART interface to implement the Bluetooth function. In some embodiments, the audio module 170 can transmit audio signals to the wireless communication module 160 through the UART interface to implement the function of playing music through a Bluetooth headset.

[0052] The MIPI interface can be used to connect the processor 110 to peripheral devices such as the display screen 194 and the camera 193. The MIPI interface includes a camera serial interface (CSI), a display serial interface (DSI), etc. In some embodiments, the processor 110 and the camera 193 communicate through the CSI interface to implement the shooting function of the electronic device 100. The processor 110 and the display screen 194 communicate through the DSI interface to implement the display function of the electronic device 100.

[0053] The GPIO interface can be configured through software. The GPIO interface can be configured as a control signal or a data signal. In some embodiments, the GPIO interface can be used to connect the processor 110 to the camera 193, the display screen 194, the wireless communication module 160, the audio module 170, the sensor module 180, etc. The GPIO interface can also be configured as an I2C interface, an I2S interface, a UART interface, a MIPI interface, etc.

[0054] The USB interface 130 is an interface that conforms to the USB standard specification and can specifically be a Mini USB interface, a Micro USB interface, a USB Type C interface, etc. The USB interface 130 can be used to connect a charger to charge the electronic device 100, and can also be used for data transmission between the electronic device 100 and peripheral devices. It can also be used to connect headphones to play audio. This interface can also be used to connect other electronic devices, such as AR devices, etc.

[0055] It can be understood that the interface connection relationships among the modules illustrated in the embodiments of the present invention are only illustrative descriptions and do not constitute a structural limitation on the electronic device 100. In other embodiments of the present application, the electronic device 100 may also adopt different interface connection manners in the above embodiments, or a combination of multiple interface connection manners.

[0056] The charging management module 140 is configured to receive a charging input from a charger. Wherein, the charger may be a wireless charger or a wired charger. In some embodiments of wired charging, the charging management module 140 may receive the charging input of the wired charger through the USB interface 130. In some embodiments of wireless charging, the charging management module 140 may receive the wireless charging input through the wireless charging coil of the electronic device 100. While charging the battery 142, the charging management module 140 may also supply power to the electronic device through the power management module 141.

[0057] The power management module 141 is configured to connect to the battery 142, the charging management module 140, and the processor 110. The power management module 141 receives the inputs from the battery 142 and / or the charging management module 140, and supplies power to the processor 110, the internal memory 121, the external memory, the display screen 194, the camera 193, the wireless communication module 160, etc. The power management module 141 may also be used to monitor parameters such as the battery capacity, the number of battery cycles, and the battery health status (leakage, impedance). In some other embodiments, the power management module 141 may also be disposed in the processor 110. In other embodiments, the power management module 141 and the charging management module 140 may also be disposed in the same device.

[0058] The wireless communication function of the electronic device 100 may be implemented through the antenna 1, the antenna 2, the mobile communication module 150, the wireless communication module 160, the modulation and demodulation processor, and the baseband processor, etc.

[0059] The antenna 1 and the antenna 2 are used to transmit and receive electromagnetic wave signals. Each antenna in the electronic device 100 may be used to cover a single or multiple communication frequency bands. Different antennas may also be multiplexed to improve the utilization rate of the antennas. For example: the antenna 1 may be multiplexed as the diversity antenna of the wireless local area network. In some other embodiments, the antenna may be used in combination with a tuning switch.

[0060] The mobile communication module 150 may provide a solution for wireless communication including 2G / 3G / 4G / 5G, etc., which is applied to the electronic device 100. The mobile communication module 150 may include at least one filter, switch, power amplifier, low noise amplifier (LNA), etc. The mobile communication module 150 may receive electromagnetic waves through the antenna 1, filter and amplify the received electromagnetic waves, and then transmit them to the modulation and demodulation processor for demodulation. The mobile communication module 150 may also amplify the signal modulated by the modulation and demodulation processor and convert it into electromagnetic waves through the antenna 1 for radiation. In some embodiments, at least some functional modules of the mobile communication module 150 may be provided in the processor 110. In some embodiments, at least some functional modules of the mobile communication module 150 and at least some modules of the processor 110 may be provided in the same device.

[0061] The modulation and demodulation processor may include a modulator and a demodulator. Among them, the modulator is used to modulate the low-frequency baseband signal to be transmitted into a medium-high frequency signal. The demodulator is used to demodulate the received electromagnetic wave signal into a low-frequency baseband signal. Subsequently, the demodulator transmits the demodulated low-frequency baseband signal to the baseband processor for processing. After being processed by the baseband processor, the low-frequency baseband signal is transmitted to the application processor. The application processor outputs a sound signal through an audio device (not limited to the speaker 170A, receiver 170B, etc.), or displays an image or video through the display screen 194. In some embodiments, the modulation and demodulation processor may be an independent device. In other embodiments, the modulation and demodulation processor may be independent of the processor 110 and be provided in the same device as the mobile communication module 150 or other functional modules.

[0062] The wireless communication module 160 may provide solutions for wireless communications applied to the electronic device 100, including wireless local area networks (WLANs) (such as wireless fidelity (Wi-Fi) networks), Bluetooth (BT), global navigation satellite systems (GNSS), frequency modulation (FM), near field communication (NFC), infrared (IR), and the like. The wireless communication module 160 may be one or more devices integrating at least one communication processing module. The wireless communication module 160 receives electromagnetic waves via the antenna 2, performs frequency modulation and filtering processing on the electromagnetic wave signals, and sends the processed signals to the processor 110. The wireless communication module 160 may also receive signals to be sent from the processor 110, perform frequency modulation and amplification on them, and convert them into electromagnetic waves through the antenna 2 for radiation.

[0063] In some embodiments, antenna 1 of electronic device 100 is coupled to mobile communication module 150, and antenna 2 is coupled to wireless communication module 160, such that electronic device 100 can communicate with a network and other devices through wireless communication technologies. The wireless communication technologies may include global system for mobile communications (GSM), general packet radio service (GPRS), code division multiple access (CDMA), wideband code division multiple access (WCDMA), time-division code division multiple access (TD-SCDMA), long term evolution (LTE), BT, GNSS, WLAN, NFC, FM, and / or IR technology, etc. The GNSS may include global positioning system (GPS), global navigation satellite system (GLONASS), beidou navigation satellite system (BDS), quasi-zenith satellite system (QZSS), and / or satellite based augmentation systems (SBAS).

[0064] Electronic device 100 implements a display function through a GPU, display screen 194, and an application processor, etc. The GPU is a microprocessor for image processing, and is connected to display screen 194 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. Processor 110 may include one or more GPUs, which execute program instructions to generate or change display information.

[0065] The display screen 194 is used to display images, videos, etc. The display screen 194 includes a display panel. The display panel can be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), a MiniLED, a MicroLED, a Micro-OLED, a quantum dot light-emitting diode (QLED), etc. In some embodiments, the electronic device 100 may include one or N display screens 194, where N is a positive integer greater than 1.

[0066] The electronic device 100 can implement the shooting function through the ISP, the camera 193, the video codec, the GPU, the display screen 194, the application processor, etc.

[0067] The ISP is used to process the data fed back by the camera 193. For example, when taking a photo, the shutter is opened, and light is transmitted through the lens to the camera photosensitive element, where the optical signal is converted into an electrical signal. The camera photosensitive element transmits the electrical signal to the ISP for processing and converts it into an image visible to the naked eye. The ISP can also perform algorithm optimization on the noise, brightness, and skin color of the image. The ISP can also optimize parameters such as the exposure and color temperature of the shooting scene. In some embodiments, the ISP can be set in the camera 193.

[0068] The camera 193 is used to capture static images or videos. An object generates an optical image through the lens and projects it onto the photosensitive element. The photosensitive element can be a charge-coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the optical signal into an electrical signal, and then transmits the electrical signal to the ISP to convert it into a digital image signal. The ISP outputs the digital image signal to the DSP for processing. The DSP converts the digital image signal into an image signal in standard RGB, YUV, etc. formats. In some embodiments, the electronic device 100 may include one or N cameras 193, where N is a positive integer greater than 1.

[0069] The digital signal processor is used to process digital signals. In addition to processing digital image signals, it can also process other digital signals. For example, when the electronic device 100 selects a frequency point, the digital signal processor is used to perform Fourier transform on the frequency point energy, etc.

[0070] The video codec is used to compress or decompress digital videos. The electronic device 100 can support one or more video codecs. In this way, the electronic device 100 can play or record videos in multiple encoding formats, such as: Moving Picture Experts Group (MPEG) 1, MPEG2, MPEG3, MPEG4, etc.

[0071] The NPU is a neural-network (NN) computing processor. By learning from the structure of biological neural networks, such as learning from the transmission pattern between human brain neurons, it can quickly process the input information and can also continuously self-learn. Through the NPU, applications such as intelligent cognition of the electronic device 100 can be realized, such as: image recognition, face recognition, speech recognition, text understanding, etc.

[0072] The external memory interface 120 can be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the electronic device 100. The external memory card communicates with the processor 110 through the external memory interface 120 to achieve the data storage function. For example, files such as music and videos are saved in the external memory card.

[0073] The internal memory 121 can be used to store computer-executable program code, and the executable program code includes instructions. The processor 110 executes various functional applications and data processing of the electronic device 100 by running the instructions stored in the internal memory 121. The internal memory 121 can include a program storage area and a data storage area. Among them, the program storage area can store the operating system, application programs required for at least one function (such as the sound playback function, the image playback function, etc.). The data storage area can store the data created during the use of the electronic device 100 (such as audio data, phone book, etc.). In addition, the internal memory 121 can include high-speed random access memory, and can also include non-volatile memory, such as at least one disk storage device, flash memory device, universal flash storage (UFS), etc.

[0074] The electronic device 100 can implement audio functions through the audio module 170, speaker 170A, receiver 170B, microphone 170C, headphone jack 170D, and the application processor, etc. For example, music playback, recording, etc.

[0075] The audio module 170 is used to convert digital audio information into an analog audio signal for output, and is also used to convert an analog audio input into a digital audio signal. The audio module 170 can also be used for encoding and decoding audio signals. In some embodiments, the audio module 170 can be disposed in the processor 110, or some functional modules of the audio module 170 can be disposed in the processor 110.

[0076] The speaker 170A, also known as the "loudspeaker", is used to convert an audio electrical signal into a sound signal. The electronic device 100 can listen to music or hands-free calls through the speaker 170A.

[0077] The receiver 170B, also known as the "earpiece", is used to convert an audio electrical signal into a sound signal. When the electronic device 100 answers a call or a voice message, the voice can be listened to by placing the receiver 170B close to the human ear.

[0078] The microphone 170C, also known as the "microphone" or "transmitter", is used to convert a sound signal into an electrical signal. When making a call or sending a voice message, the user can speak by placing the mouth close to the microphone 170C to input the sound signal into the microphone 170C. The electronic device 100 can be provided with at least one microphone 170C. In some other embodiments, the electronic device 100 can be provided with two microphones 170C, which can not only collect sound signals but also implement a noise reduction function. In some other embodiments, the electronic device 100 can also be provided with three, four or more microphones 170C, which can collect sound signals, reduce noise, identify the sound source, and implement functions such as directional recording.

[0079] The headphone jack 170D is used to connect a wired headphone. The headphone jack 170D can be a USB interface 130, or a 3.5 mm open mobile terminal platform (OMTP) standard interface, or a cellular telecommunications industry association of the USA (CTIA) standard interface.

[0080] The pressure sensor 180A is used to sense pressure signals and can convert pressure signals into electrical signals. In some embodiments, the pressure sensor 180A may be disposed on the display screen 194. There are many types of pressure sensors 180A, such as resistive pressure sensors, inductive pressure sensors, capacitive pressure sensors, etc. The capacitive pressure sensor may include at least two parallel plates having conductive materials. When a force acts on the pressure sensor 180A, the capacitance between the electrodes changes. The electronic device 100 determines the intensity of the pressure according to the change in capacitance. When a touch operation acts on the display screen 194, the electronic device 100 detects the intensity of the touch operation according to the pressure sensor 180A. The electronic device 100 can also calculate the position of the touch according to the detection signal of the pressure sensor 180A. In some embodiments, touch operations acting on the same touch position but with different touch operation intensities may correspond to different operation instructions. For example: when a touch operation with a touch operation intensity less than the first pressure threshold acts on the short message application icon, the instruction to view the short message is executed. When a touch operation with a touch operation intensity greater than or equal to the first pressure threshold acts on the short message application icon, the instruction to create a new short message is executed.

[0081] The gyroscope sensor 180B can be used to determine the motion posture of the electronic device 100. In some embodiments, the angular velocity of the electronic device 100 around three axes (i.e., the x, y, and z axes) can be determined by the gyroscope sensor 180B. The gyroscope sensor 180B can be used for anti-shake shooting. Exemplarily, when the shutter is pressed, the gyroscope sensor 180B detects the angle of jitter of the electronic device 100, calculates the distance that the lens module needs to compensate according to the angle, and makes the lens offset the jitter of the electronic device 100 through reverse movement to achieve anti-shake. The gyroscope sensor 180B can also be used for navigation and somatosensory game scenarios.

[0082] The barometric pressure sensor 180C is used to measure barometric pressure. In some embodiments, the electronic device 100 calculates the altitude according to the barometric pressure value measured by the barometric pressure sensor 180C to assist in positioning and navigation.

[0083] The magnetic sensor 180D includes a Hall sensor. The electronic device 100 can use the magnetic sensor 180D to detect the opening and closing of the flip leather case. In some embodiments, when the electronic device 100 is a flip phone, the electronic device 100 can detect the opening and closing of the flip according to the magnetic sensor 180D. Furthermore, according to the detected opening and closing state of the leather case or the opening and closing state of the flip, features such as automatic flip unlocking are set.

[0084] The acceleration sensor 180E can detect the magnitude of the acceleration of the electronic device 100 in various directions (generally three axes). When the electronic device 100 is stationary, the magnitude and direction of gravity can be detected. It can also be used to identify the posture of the electronic device and is applied to applications such as horizontal and vertical screen switching and pedometers.

[0085] A distance sensor 180F for measuring distance. The electronic device 100 can measure distance by infrared or laser. In some embodiments, when shooting a scene, the electronic device 100 can use the distance sensor 180F to measure distance for fast focusing.

[0086] The proximity light sensor 180G may include, for example, a light emitting diode (LED) and a light detector, such as a photodiode. The light emitting diode may be an infrared light emitting diode. The electronic device 100 emits infrared light outward through the light emitting diode. The electronic device 100 uses the photodiode to detect the infrared reflected light from nearby objects. When sufficient reflected light is detected, it can be determined that there is an object near the electronic device 100. When insufficient reflected light is detected, the electronic device 100 can determine that there is no object near the electronic device 100. The electronic device 100 can use the proximity light sensor 180G to detect when the user holds the electronic device 100 close to the ear for a call, so as to automatically turn off the screen to save power. The proximity light sensor 180G can also be used for automatic unlocking and locking of the holster mode and pocket mode.

[0087] The ambient light sensor 180L is used to sense the ambient light brightness. The electronic device 100 can adaptively adjust the brightness of the display screen 194 according to the sensed ambient light brightness. The ambient light sensor 180L can also be used to automatically adjust the white balance when taking pictures. The ambient light sensor 180L can also cooperate with the proximity light sensor 180G to detect whether the electronic device 100 is in the pocket to prevent accidental touch.

[0088] The fingerprint sensor 180H is used to collect fingerprints. The electronic device 100 can use the collected fingerprint characteristics to achieve fingerprint unlocking, access application locks, fingerprint photography, fingerprint answering calls, etc.

[0089] The temperature sensor 180J is used to detect temperature. In some embodiments, the electronic device 100 uses the temperature detected by the temperature sensor 180J to execute a temperature processing strategy. For example, when the temperature reported by the temperature sensor 180J exceeds a threshold, the electronic device 100 reduces the performance of the processor near the temperature sensor 180J to reduce power consumption and implement thermal protection. In other embodiments, when the temperature is lower than another threshold, the electronic device 100 heats the battery 142 to avoid abnormal shutdown of the electronic device 100 caused by low temperature. In some other embodiments, when the temperature is lower than yet another threshold, the electronic device 100 boosts the output voltage of the battery 142 to avoid abnormal shutdown caused by low temperature.

[0090] The touch sensor 180K, also known as the "touch panel". The touch sensor 180K can be disposed on the display screen 194. The touch sensor 180K and the display screen 194 together form a touch screen, also known as the "touch display screen". The touch sensor 180K is used to detect touch operations acting thereon or in its vicinity. The touch sensor can transmit the detected touch operation to the application processor to determine the type of touch event. Visual output related to the touch operation can be provided through the display screen 194. In some other embodiments, the touch sensor 180K can also be disposed on the surface of the electronic device 100, at a position different from that of the display screen 194.

[0091] The bone conduction sensor 180M can acquire vibration signals. In some embodiments, the bone conduction sensor 180M can acquire the vibration signals of the vibrating bone mass of the human vocal part. The bone conduction sensor 180M can also contact the human pulse to receive blood pressure pulsation signals. In some embodiments, the bone conduction sensor 180M can also be disposed in the earphone to form a bone conduction earphone. The audio module 170 can parse out voice signals based on the vibration signals of the vibrating bone mass of the human vocal part acquired by the bone conduction sensor 180M to implement the voice function. The application processor can parse out heart rate information based on the blood pressure pulsation signals acquired by the bone conduction sensor 180M to implement the heart rate detection function.

[0092] The button 190 includes a power-on button, a volume button, etc. The button 190 can be a mechanical button or a touch button. The electronic device 100 can receive button inputs to generate key signal inputs related to the user settings and function controls of the electronic device 100.

[0093] The motor 191 can generate vibration prompts. The motor 191 can be used for incoming call vibration prompts and also for touch vibration feedback. For example, touch operations on different applications (such as taking pictures, playing audio, etc.) can correspond to different vibration feedback effects. Touch operations on different regions of the display screen 194 can also cause the motor 191 to correspond to different vibration feedback effects. Different application scenarios (such as time reminder, receiving messages, alarm clock, games, etc.) can also correspond to different vibration feedback effects. The touch vibration feedback effect can also support customization.

[0094] The indicator 192 can be an indicator light and can be used to indicate the charging state, power change, and can also be used to indicate messages, missed calls, notifications, etc.

[0095] The SIM card interface 195 is used to connect to the SIM card. The SIM card can be inserted into or removed from the SIM card interface 195 to achieve contact and separation from the electronic device 100. The electronic device 100 can support 1 or N SIM card interfaces, where N is a positive integer greater than 1. The SIM card interface 195 can support Nano SIM cards, Micro SIM cards, SIM cards, etc. Multiple cards can be inserted into the same SIM card interface 195 simultaneously. The types of the multiple cards can be the same or different. The SIM card interface 195 can also be compatible with different types of SIM cards. The SIM card interface 195 can also be compatible with external memory cards. The electronic device 100 interacts with the network through the SIM card to implement functions such as calls and data communication. In some embodiments, the electronic device 100 uses an eSIM, that is, an embedded SIM card. The eSIM card can be embedded in the electronic device 100 and cannot be separated from the electronic device 100.

[0096] Next, some terms and related technologies involved in this application are explained to facilitate understanding by those skilled in the art.

[0097] 1. BERT (Bidirectional Encoder Representation from Transformers) model: The BERT model is the encoder of the bidirectional transformer. Among them, the transformer is a method that fully relies on self-attention to calculate the input and output representations. BERT uses the masked model to achieve the bidirectionality of the language model, proving the importance of bidirectionality for language representation pre-training. The BERT model is a truly bidirectional language model, and each word can simultaneously utilize the context information of that word. BERT aims to pre-train deep bidirectional representations by jointly adjusting the context in all layers. Therefore, the pre-trained BERT representation can be fine-tuned through an additional output layer for constructing state-of-the-art models suitable for a wide range of tasks.

[0098] After adding a fully connected layer to the BERT model and training it, the BERT model after removing the fully connected layer can be used for various natural language processing tasks (including sequence labeling tasks, classification tasks, sentence relationship judgment, and generative tasks). The following introduces how to train the BERT model with a fully connected layer added through two pre-training tasks.

[0099] Task 1: Generative task

[0100] Method: Randomly mask some words in a sentence (for example, by replacing them with a uniform marker [MASK]). The sentence is then fed into the BERT model, which predicts the masked words. The bidirectional language model is then trained by predicting these masked words, and each word's representation is contextually informed. Assuming the second word in the sentence is masked, its corresponding vector representation is fed into the BERT model to predict the masked word. The subsequent loss is calculated only for the masked portion.

[0101] Task 2: Sentence Relationship Judgment

[0102] Method: First, we get a pair of sentences belonging to the context, that is, two sentences. Then we need to add some special tokens to these two consecutive sentences. That is, add a [cls] at the beginning of the sentence, and add [sep] between the two sentences and at the end of the sentence. Specifically, it is as follows:

[0103] The two sentences are [cls]my dog is cute[sep]he likes playing[sep] and [cls]my dog is cute[sep]he likes playing[sep]. We also need to prepare two sentences of the same format, but without a contextual relationship: [cls]my dog is cute[sep]penguins are not good at flying[sep]. This is a case where the upper and lower sentences do not have a contextual relationship. In actual training, we set the ratio of the two cases to be 1:1, meaning that half of the training text in the training data has a contextual relationship, and the other half does not.

[0104] The corresponding vector representation is input into the BERT model, and the fully connected layer outputs the detection result to predict whether the two sentences belong to the context relationship. The bidirectional language model is then trained by predicting whether the two sentences belong to the context relationship.

[0105] 2. Few-shot learning: Few-shot learning is the application of meta-learning to supervised learning. Meta-learning, also known as learning to learn, aims to leverage previously acquired knowledge to solve new problems. During the meta-training phase, the dataset is broken down into different meta-tasks to learn the model's generalization capabilities under changing categories. During the meta-testing phase, classification can be completed for completely new categories without modifying the existing model.

[0106] For example, humans are very good at recognizing a new object through a very small number of samples. For instance, a young child can learn what a "zebra" and a "rhinoceros" are just from some pictures in a book. Inspired by humans' fast learning ability, researchers hope that after a machine learning model has learned a large amount of data of a certain category, for a new category, it can quickly learn with only a small number of samples. This is the problem that few-shot learning aims to solve.

[0107] 3. An artificial intelligence (AI) model is a type of machine learning model, which is essentially a mathematical model consisting of a large number of parameters and mathematical formulas (or mathematical rules). Its purpose is to learn some mathematical expressions that can provide the correlation between the input value x and the output value y. The mathematical expression that can provide the correlation between x and y is the trained AI model. Generally, using some historical data (i.e., x and y), the AI model obtained by training the initial AI model can be used to obtain a new y based on a new x, thereby achieving predictive analysis. The process of predictive analysis is also called inference.

[0108] 4. Supervised learning is the process of training an initial AI model using multiple training data carrying annotation information. Each training data serves as the input data for the initial AI model, and the annotation information of each training data is the expected output of the initial AI model. The training process is to input the training data into the initial AI model. The initial AI model performs a series of mathematical calculations on the input data to obtain an output result, and then compares the output result with the annotation information of the training data, adjusts the parameters in the initial AI model, and iteratively trains the initial AI model in turn to make the result output by the initial AI model closer to the annotation information corresponding to the input training data. The trained AI model can be used to predict the results of unknown data. For example, the training data can be a sample annotation sequence, and the annotation information carried by each sample annotation sequence is the label of the sample annotation sequence (such as a location). Then, the sample annotation sequences carrying annotation information are input into the initial AI model for training in turn. After each training, the label of the sample annotation sequence output by the initial AI model is compared with the label of the sample annotation sequence in the corresponding annotation information, and the parameters in the initial AI model are adjusted, and iterative training is continuously carried out to make the label of the sample annotation sequence of the target detection box output by the initial AI model close to the label of the sample annotation sequence in the annotation information. In this way, the initial AI model is trained, that is, the trained AI model has the ability to label the unlabeled sequence.

[0109] 5. Viterbi Decoding: The Viterbi decoding algorithm is essentially maximum likelihood decoding. It utilizes the special structure of the coding network graph, thereby reducing the complexity of path calculation. This algorithm includes: calculating the similarity (or distance) between the paths from time t to each state on the trellis graph and the received sequence, and removing the paths on the trellis graph that cannot be the object of maximum likelihood selection. That is, if there are two paths reaching the same state, the path with the best metric is selected as the surviving path, and other paths are discarded. By performing such selection operations on all states, discarding impossible paths earlier can greatly reduce the complexity of the decoder.

[0110] The Viterbi decoder mainly includes three functional units:

[0111] (1) Branch Metric Unit: Calculate the similarity (or path metric) of each branch in the trellis graph relative to the received sequence according to the signal sequence received at time t and the output sequence of each branch. i

[0112] (2) Add-Compare-Select Unit: Calculate the path metric from the state at time t to the state at time t according to the distance metric result of each branch calculated by the branch metric unit at time t and the path metric of the state at time t. Since at time t, each state may have multiple input paths, the add-compare-select unit performs path merging according to the branch path metric values input to each state at time t, and selects the path with the smallest branch path metric as the surviving path of this state at time t, and discards other branch paths. i i-1 i-1 i i i i

[0113] (3) Surviving Path Register Unit: Store the decoding results on the surviving path, and only when the path metric calculation reaches a deeper part of the trellis graph, the first decoded output is generated (select the decoding result corresponding to the minimum value of the surviving path of each state for output).

[0114] 6. Intents and Slots:

[0115] 6.1 Definition of Intents and Slots

[0116] An intent refers to what the electronic device recognizes as the actual or potential needs of the user. Fundamentally speaking, an intent is a classifier that classifies user needs into a certain type.

[0117] The intent and slots together constitute the "user action". Since electronic devices cannot directly understand natural language, the role of intent recognition is to map natural language or operations into a structured semantic representation that machines can understand. For example, for the sentence "What's the weather like today", its intent is "inquiring about the weather". One intent may correspond to several slots. For example, when asking about the bus route, necessary parameters such as the departure location, destination, and time need to be given. The above parameters are the slots corresponding to the intent of "inquiring about the bus route".

[0118] The main goal of the slot filling task for text sequences is to extract the slot values of the predefined semantic slots in the semantic frame from the input text sequence on the premise of knowing the semantic frame of a specific domain or specific intent. The semantic slot filling task can be transformed into a sequence labeling task, that is, using the classic BIO tagging method. Among them, "B" is used to mark that a certain word is the beginning (begin) of a certain semantic slot, "I" is used to label that a word is the continuation (inside) of a certain semantic slot, and "O" indicates that a certain word is outside the semantic slot (outside). Using the BIO tagging method to label an unlabeled text sequence, each word in the labeled text sequence will carry a labeling tag.

[0119] Taking a specific requirement of "inquiring about the weather" as an example, the slot filling based on intent recognition involved in the embodiments of the present application is introduced:

[0120] Text sequence: "What's the weather like in Shanghai today";

[0121] Intent: Inquiring about the weather;

[0122] Slots: Slot 1: Time; Slot 2: Location.

[0123] Slot values: Time slot value: Today; Location slot value: Shanghai.

[0124] Figure 2 This is a schematic diagram of the relationship between an intent and slots in the embodiments of the present application. As Figure 2 shown, in this example, two necessary slots are defined for the "inquiring about the weather" task, which are "Time" and "Location" respectively.

[0125] 6.2. Slot Filling

[0126] After defining the intent and slots, slot filling can be performed on the text sequence according to the user intent of the text sequence and the predefined slots.

[0127] The task of slot filling is to extract information from the text sequence and fill it into the predefined slots. For example, in Figure 2The intents and corresponding slots have been defined. For the text sequence "What's the weather like in Shanghai today", the slot extraction device can extract "today" and "Shanghai" and fill them into the "time" slot and "location" slot respectively.

[0128] This application provides a text annotation extraction method. The server can receive a first annotation request sent by an electronic device; in response to the first annotation request, the server starts a dynamic annotation service, where the dynamic annotation service corresponds to a first dynamic annotation model, and the first dynamic annotation model is trained with sample text sequences with a first label; the server receives a first text sequence sent by the electronic device and first annotation information corresponding to the first text sequence, where the first text sequence includes first text information and a second label, and the first annotation information is used to indicate the position of the text with the attribute of the second label in the first text sequence, and the first label and the second label may be different; the server obtains a second text sequence; when the number of the first text sequences is greater than a first threshold, the server determines second annotation information of the second text sequence based on the first text sequence and the first annotation information through the first dynamic annotation model, and the second annotation information is used to indicate the position of the text with the attribute of the second label in the second text sequence; the server extracts the text with the attribute of the second label in the second text sequence based on the second annotation information. Through this method, when the annotation label of the automatic annotation model changes, there is no need to retrain the model, which can improve the efficiency and accuracy of automatic annotation.

[0129] The type of the first label can include any one, several or other slot labels such as time, origin, destination, vehicle type, etc., and the type of the second label can include any one or several of time, origin, destination, vehicle type, etc. This application does not limit the types of the first label and the second label.

[0130] The embodiments of this application can also be applied to annotating multiple second labels of the first text sequence. For example, simultaneously annotating the time and origin labels of the first text sequence; or simultaneously annotating the time, origin label and destination label of the first text sequence, etc. This application does not limit the number of types of labels annotated for the first text sequence.

[0131] As Figure 3 shown, Figure 3 is the system architecture diagram of this application. The system includes an electronic device 100 and a server 200.

[0132] Among them, the electronic device 100 can receive task items created by the user. The electronic device 100 provides operation options for the user according to the task items, and the operation options can include selecting the dynamic annotation model service 3101 and selecting the automatic annotation model service 3102.

[0133] Among them, the dynamic annotation model service 3101 corresponds to a dynamic annotation model 3103 (the first dynamic annotation model). The dynamic annotation model service 3101 can be used to label the slot tags (the second tags) of the unlabeled text sequence through the dynamic annotation model 3103 when the inference platform 310 does not have an automatic annotation model service 3102, and extract the slot values of the unlabeled text sequence according to the slot tags of the unlabeled text sequence.

[0134] The dynamic annotation model 3103 can be used to label the preset tags of the unlabeled text sequence. Among them, the dynamic annotation model 3103 can be an annotation model pre-trained through a large number of text sequences with preset tags (the first tags). The preset tag system for the dynamic annotation model 3103 to label the text sequence can be different from the slot tag system of the task data to be created currently. For example: the annotation data of the dynamic annotation model 3103 is data in the taxi field, and the preset tag system contains the departure place and the destination. The slot tag system of the current annotation task needs to add a tag type. For example, on the basis that the preset tag system contains the departure place and the destination, add tags of vehicle types (taxi, express car, private car, etc.) to obtain the slot tag system. The slot tag system contains tags for the departure place, tags for the destination, and tags for vehicle types (taxi, express car, private car, etc.).

[0135] The automatic annotation model service 3102 corresponds to an automatic annotation model 3104. The automatic annotation model service 3102 can be used to label the preset tags of the unlabeled text sequence through the automatic annotation model 3104 according to the preset tags.

[0136] The automatic annotation model 3104 is an annotation model pre-trained through a large number of text sequences with preset tags. The automatic annotation model 3104 can only label the preset tags of the unlabeled text sequence. Exemplarily, if the automatic annotation model 3104 is trained through data containing tags of vehicle types (taxi, express car, private car, etc.), the automatic annotation model 3104 can only label the tags of vehicle types (taxi, express car, private car, etc.) of the unlabeled text sequence. If the user creates a new task item (such as labeling the destination tag of the unlabeled text sequence), the automatic annotation model 3104 needs to be re-trained through text sequences with destination tags.

[0137] If there is an automatic annotation model for the current text sequence annotation task in the operation options, the electronic device 100 receives the user's selection operation to start the automatic annotation model service 3102; if there is no automatic annotation model in the inference platform 310, or the preset tags of the automatic annotation model do not match the slot tags of the current text sequence, the electronic device 100 receives the user's selection operation to start the dynamic annotation model service 3101.

[0138] In response to the user's operation of selecting the dynamic annotation model service 3101, the electronic device 100 sends a request to start the dynamic annotation model service 3101 to the server 200, and the server 200 will start the dynamic annotation model service 3101 to annotate the slot labels of the unannotated text sequence. Exemplarily, taking the current annotation task of annotating the "time" and "destination" slot labels in the unannotated text sequence as an example, the execution process of the dynamic annotation model service 3101 can be divided into the following steps:

[0139] Step 1: The server 200 obtains a text sequence with slot labels (the first text sequence).

[0140] Among them, the slot label is the label of the text sequence that needs to be annotated in the current annotation task. For example, the slot label can be a label for "destination".

[0141] Exemplarily, the text sequence can be "I'm going to Beijing today".

[0142] The slot labels are "time" and "destination".

[0143] The text sequence with slot labels can be expressed as ""I - O"" ""Today - B - time"" ""day - I - time"" ""going to - O"" ""Beijing - B - destination"" ""city - I - destination""".

[0144] Step 2: The server 200 determines whether the number of text sequences with slot labels has reached a threshold (the first threshold).

[0145] Among them, the threshold can be custom - set according to actual needs. For example, if the number of text sequences with slot labels reaches n (n is a positive integer greater than or equal to 1), step 3 is executed. If the number of text sequences with slot labels has not reached the threshold, return to step 1, and the dynamic annotation model service 3101 continues to obtain text sequences with slot labels until the server 200 determines that the number of text sequences with slot labels has reached the threshold.

[0146] Step 3: When the server 200 determines that the number of text sequences with slot labels has reached the threshold, the server 200 updates the dynamic annotation model service 3101.

[0147] Before the update, the dynamic annotation model service 3101 can be used to annotate the unannotated text sequence with a preset label (e.g., "time") through the dynamic annotation model 3103. The server 200 determines that the number of text sequences with slot labels has reached a threshold, and the server 200 updates the dynamic annotation model service 3101. After the update, the dynamic annotation model service 3101 can be used to annotate the unannotated text sequence with slot labels (e.g., "time" and "destination") through the dynamic annotation model 3103. The preset label and the slot label are of different label types.

[0148] Step 4: The dynamic annotation model 3103 begins to annotate the slot labels (eg, "time" and "destination") of the unannotated text sequence (the second text sequence).

[0149] The dynamic labeling model 3103 begins labeling the slot labels of the unlabeled text sequence and extracts the slot values of the text sequence according to the slot labels.

[0150] The following describes in detail how the dynamic labeling model 3103 labels the slot labels of the unlabeled text sequence and extracts the slot values of the text sequence based on the slot labels.

[0151] Next, we will introduce in detail how the dynamic annotation model 3103 annotates the slot labels of the unlabeled text sequence and obtains the slot value of the unlabeled text sequence according to the slot labels of the text sequence. Figure 4 As shown, Figure 4 Flowchart for labeling slots of unlabeled text sequences for the dynamic labeling model 3103.

[0152] Step 1: The dynamic tagging model 3103 calculates the transition probability between the tagging tags in the text sequence based on the text sequence with slot tags.

[0153] In the present application, the transfer matrix is modeled as a non-parametric method based on a method of mathematical statistics. Specifically, the transfer matrix here is obtained by calculating the number of transfers between the annotation labels in the text sequence based on the text sequence with slot labels. At the same time, in order to avoid errors caused by too little data, the dynamic annotation model 3103 can use the Laplace smoothing constant to smooth the observations on the transfer matrix. The advantage of modeling in this way is that the transfer matrix is non-parametric, can be dynamically adjusted with the text sequence with slot labels, and can adapt to transfers under different system labels. According to the law of large numbers, as the number of text sequences with slot labels increases, the error of the transfer matrix will gradually decrease.

[0154] The following describes how the dynamic tagging model 3103 calculates the transition probability between tagging tags in a text sequence based on a text sequence with slot tags.

[0155] Suppose the number of text sequences with tags is n. The types of slot tags are m.

[0156] First, the dynamic annotation model 3103 calculates the size of the transition matrix. The size of the transition matrix is t * t, where t = 2 * m + 1.

[0157] If the text sequence is annotated in the "BIO" annotation method, then t = 1 "O" tag + m "B"-starting tags + m "I"-starting tags.

[0158] Exemplarily, the text sequence can be "Buy a ticket to Shanghai tomorrow".

[0159] The slot tags are "time" and "destination".

[0160] The annotation result of the text sequence can be expressed as "Buy - O" "tomorrow - B - time" "day - I - time" "go - O" "Shanghai - B - destination" "sea - I - destination" "of - O" "plane - O" "ticket - O".

[0161] Then the number of types m of the slot tags of this text sequence is 2.

[0162] As Figure 5 shown, Figure 5 is a schematic diagram of the annotation result of the text sequence.

[0163] As Figure 6 shown, Figure 6 is Figure 5 a schematic diagram of the first transition matrix corresponding to the text sequence with slot tags as shown.

[0164] For a text sequence with slot tags, the number of transitions from the annotation tag corresponding to each word in the text sequence to the annotation tag corresponding to each word can only occur between two adjacent words, and the number of transitions only calculates the transitions from the first word to the second word among two adjacent words.

[0165] For example, in Figure 6In the transition matrix shown, the value "2" in the first row and first column indicates that there has been one transfer from label "的-0" to label "机-0" and one transfer from label "机-0" to label "票-0". This means that according to the rule, there have been two transfers from label "0" to label "0". The value "1" in the second row and second column indicates that there has been one transfer from label "买-0" to label "明-B-时间". This means that according to the rule, there has been one transfer from label "0" to label "B-时间". The value "0" in the third row and third column indicates that there has been no transfer from label "0" to label "I-时间" according to the rule. The value "1" in the fourth row and fourth column indicates that there has been one transfer from label "去-0" to label "上-B-Destination". This means that according to the rule, there has been one transfer from label "0" to label "B-Destination". The value "0" in the fifth row and fifth column indicates that there has been no transfer from label "0" to label "I-Destination" according to the rule.

[0166] Similarly, the dynamic tagging model 3103 calculates the number of transitions between the tagging labels corresponding to each word in the tagged data, and obtains Figure 6 The first transfer matrix is shown.

[0167] After obtaining the transfer matrix, in order to avoid errors due to too little data, the dynamic labeling model 3103 can use the Laplace smoothing constant to smooth the observation values of the first transfer matrix.

[0168] The Laplace smoothing constant is the illegal position transfer matrix. In the illegal position transfer matrix, the value of the illegal position is 0, and the value of the legal position is 1. In the text sequence, the illegal position is that the starting word of the first label cannot be adjacent to the continuation word of the second label, and the starting word of the first label is before the continuation word of the second label; the illegal position can also be that the continuation word of the first label cannot be adjacent to the continuation word of the second label, and the continuation word of the first label is before the continuation word of the second label; the illegal position can also be that the starting word of the second label cannot be adjacent to the continuation word of the first label, and the continuation word of the first label is before the continuation word of the second label; the illegal position can also be that the continuation word of the second label cannot be adjacent to the continuation word of the first label, and the continuation word of the second label is before the continuation word of the first label; the illegal position can also be that the non-label word cannot be adjacent to the continuation word of the first label, and the non-label word is before the continuation word of the first label; the illegal position can also be that the non-label word cannot be adjacent to the continuation word of the second label, and the non-label word is before the continuation word of the second label.

[0169] like Figure 7 As shown, Figure 7 Schematic diagram of illegal position transfer matrix.

[0170] Exemplarily, for the text sequence with slot labels "Buy - O", "Ming - B - Time", "Tian - I - Time", "Go - O", "Shang - B - Destination", "Hai - I - Destination", "Of - O", "Air - O", "Ticket - O". Then the positions from label "O" to label "Tian - I - Time" and the positions from label "O" to label "Hai - I - Destination" are illegal positions.

[0171] Therefore, the number of transitions from label "O" to label "Tian - I - Time" is 0, and the number of transitions from label "O" to label "Hai - I - Destination" is 0. As Figure 8 shown. Figure 8 The transition matrix shown is a schematic diagram of the second transition matrix after the dynamic annotation model 3103 de - smooths the observed values of the first transition matrix through the Laplace smoothing constant. That is Figure 8 a schematic diagram of the second transition matrix after adding the second transition matrix and the illegal - position transition matrix.

[0172] The dynamic annotation model 3103 normalizes the second transition matrix.

[0173] Figure 9 , Figure 9 which is a schematic diagram of the normalized transition matrix. Here, normalization means dividing the value at each position in the matrix by the sum of the values in the row where the value at that position is located.

[0174] For example, in the matrix as Figure 9 shown, the number of transitions from label "O" to label "O" is 3, "3 / (3 + 2+0 + 2+0)=3 / 7", then according to the rule, the number of transitions from label "O" to label "O" after normalization is "3 / 7". After processing the values at each position in the matrix as Figure 8 shown in the above - mentioned way, we get Figure 9 the normalized transition matrix shown.

[0175] The values at each position in the normalized transition matrix represent the transition probability from the row - annotation label at that position to the column - annotation label at that position.

[0176] For example, in the normalized transition matrix as Figure 9 shown, the value in the first row and the first column is "3 / 7", and this value represents that the transition probability from label "O" to label "O" is "3 / 7".

[0177] Step 2: The dynamic annotation model 3103 calculates the emission probability of each word in the un - annotated text sequence to each annotation label according to the text sequence with slot labels.

[0178] This application uses metric-based representation learning. The number of text sequences with slot labels is dynamically adjusted, and the number of text sequences with slot labels will be increasing. Specifically, the dynamic annotation model 3103 pairs each unlabeled text sequence with a text sequence with a slot label to form the format of [CLS]A[SEP][B][pad], where A is the unlabeled text sequence and B is the text sequence with a slot label. Assuming there are n text sequences with slot labels, then n pairs of paired data can be obtained. The dynamic annotation model 3103 inputs the n pairs of paired data into the BERT model to obtain the upstream vector and the downstream vector. The dynamic annotation model 3103 calculates the representation of the label based on the downstream vector and the annotation label in the text sequence with a slot label. And the dynamic annotation model 3103 calculates the similarity between the upstream vector and the representation of the label and normalizes it to obtain the emission probability of each word in the unlabeled text sequence to each annotation label.

[0179] Specifically, the dynamic annotation model 3103 can calculate the emission probability of each word in the unlabeled text sequence to each annotation label based on the text sequence with a slot label through the following steps:

[0180] First, the dynamic annotation model 3103 obtains the upstream vector and the downstream vector of the paired data according to the BERT model.

[0181] In the first step, the dynamic annotation model 3103 pairs each unlabeled text sequence with n text sequences with slot labels to form the format of [CLS]A[SEP][B][pad], obtaining n pairs of paired data.

[0182] Exemplarily, the unlabeled text sequence can be "Buy a ticket to Shenzhen today". Among the n text sequences with slot labels, one text sequence with a slot label can be ""Buy - O"" ""Tomorrow - B - time"" ""Day - I - time"" ""Go - O"" ""Shanghai - B - destination"" ""Of - O"" ""Machine - O"" ""Ticket - O""". The obtained paired data is [CLS] ""Buy"" ""Today"" ""Go"" ""Shenzhen"" ""Of"" ""Machine"" ""Ticket""""[SEP] ""Buy"" ""Tomorrow"" ""Go"" ""Shanghai"" ""Of"" ""Machine"" ""Ticket""""[pad].

[0183] In the second step, the dynamic annotation model 3103 inputs the n pairs of paired data into the BERT model to obtain the upstream vector and the downstream vector.

[0184] Before the dynamic annotation model 3103 inputs the n pairs of paired data into the BERT model, it is necessary to process the n pairs of paired data, that is, to obtain the word vector, position vector and sentence vector of each pair of paired data.

[0185] First, the dynamic annotation model 3103 inputs each pair of paired data into the embedding layer, and the embedding layer outputs the word vectors of each pair of paired data.

[0186] For example, the paired data is [CLS] ""Buy"" ""Today"" ""Go"" ""Shenzhen"" ""Of"" ""Airplane"" ""Ticket"" ""[SEP] ""Buy"" ""Tomorrow"" ""Go"" ""Shanghai"" ""Of"" ""Airplane"" ""Ticket"" ""Padding"" [pad]. Then each word and special symbol in this paired data needs to be converted into a word vector.

[0187] Secondly, the above word vectors are input into the segmentation layer, and multiple word vectors are segmented according to the at least one symbol vector to obtain n sets of word vectors, and the n sets of word vectors correspond to n segmentation codes.

[0188] For example, the paired data is [CLS] ""Buy"" ""Today"" ""Go"" ""Shenzhen"" ""Of"" ""Airplane"" ""Ticket"" ""[SEP] ""Buy"" ""Tomorrow"" ""Go"" ""Shanghai"" ""Of"" ""Airplane"" ""Ticket"" ""Padding"" [pad]. The dynamic annotation model 3103 segments the paired data into sentence A ""Buy"" ""Today"" ""Go"" ""Shenzhen"" ""Of"" ""Airplane"" ""Ticket"" "" and sentence B ""Buy"" ""Tomorrow"" ""Go"" ""Shanghai"" ""Of"" ""Airplane"" ""Ticket"" "". Sentence A will be added with delimiter A, and sentence B will be added with delimiter B.

[0189] Through position encoding, encoding operations are performed on each delimiter to determine the position information encoding of each delimiter, and the position vectors of each word in the paired data are obtained.

[0190] Finally, the dynamic annotation model 3103 generates the sentence vectors of the paired data according to the word vectors and position vectors of each word in the paired data.

[0191] Similarly, n pairs of paired data are processed in the above manner to obtain the position vectors, word vectors and sentence vectors of each pair of paired data respectively.

[0192] The dynamic annotation model 3103 inputs the position vectors, word vectors and sentence vectors of n pairs of paired data into the BERT model, and the BERT model outputs an n-dimensional vector, and the n-dimensional vector can be represented as a matrix [batch size, seq_len, hidden].

[0193] Now there are n text sequences with slot labels, then batch size = n. hidden represents the number of hidden layers, set to x. For example, the BERT model can be the BERT-base model, then hidden = 768.

[0194] seq_len represents the length of the paired data. For each pair of paired data, there are p words in the unlabeled text sequence and m words in the text sequence with slot labels. Adding the delimiters, seq_len = p + m + 2.

[0195] Exemplarily, assume the paired data is [CLS] ""Buy"" ""Today"" ""Go"" ""To"" ""Shenzhen"" ""Of"" ""The"" ""Airplane"" ""Ticket"" ""[SEP] ""Buy"" ""Tomorrow"" ""Go"" ""To"" ""Shanghai"" ""Of"" ""The"" ""Airplane"" ""Ticket"" ""Pad, sentence A has 11 words, sentence B has 9 words, and adding the delimiters, seq_len = 22.

[0196] In the n-dimensional vector, the position corresponding to sentence A is the upstream vector, and the position corresponding to sentence B is the downstream vector.

[0197] It can be understood that the first-dimensional vector is the upstream vector and downstream vector corresponding to the first pair of paired data, the second-dimensional vector is the upstream vector and downstream vector corresponding to the second pair of paired data, and the n-dimensional vector is the upstream vector and downstream vector corresponding to the nth pair of paired data.

[0198] Secondly, the dynamic annotation model 3103 calculates the representation of the labels based on the downstream vector and the annotation labels of n text sequences with slot labels.

[0199] The calculation method of the representation of the labels is to take the average of the word vectors corresponding to each annotation label in the n text sequences with slot labels.

[0200] Specifically, for a pair of paired data, there are p words in the unlabeled text sequence, each text sequence with slot labels has m words, and there are n text sequences with slot labels in the support set. Then there are a total of m * n words, and each word has a word vector. The representation of the m * n word vectors is a matrix [m * n, x].

[0201] For the paired data, each word in each labeled data has an annotation label. Assume the number of types of annotation labels is s, and each label corresponds to a one-hot vector, which can be represented as a matrix [1, s]. Then the label vectors corresponding to the m * n words are represented as a matrix [m * n, s]. In the one-hot vector, the value at the position corresponding to the annotation label is 1, and the values at the positions of other annotation labels are 0.

[0202] Perform matrix operation on the transposed matrix of the labels corresponding to the m * n words and the matrix of the m * n word vectors, that is, [s, m * n] * [m * n, x] = [s, x]. Then each of the s labels corresponds to a vector. Divide the result by the number of times each label appears in the m * n words to obtain the representation of the labels, and the representation of the labels can be represented as a matrix [s, x].

[0203] Then, the dynamic annotation model 3103 calculates the similarity between the above-context vector and the representation of the label and normalizes it.

[0204] The dynamic annotation model 3103 calculating the similarity between the above-context vector and the representation of the label is to calculate the vector distance between the above-context vector and the representation vector of the label.

[0205] The above-context vector is the vector representation output by the BERT model for the unannotated text sequence. Sentence A has p words, and there are a total of n pairs of paired sentences. The average value of the n above-context vectors is calculated, and the above-context vector can be represented as a matrix [p, x].

[0206] The dimension of the representation matrix of the label is different from that of the above-context vector representation matrix. Therefore, the matrix needs to be extended into a matrix with the same dimension for convenient operation.

[0207] First, the representation matrix of the label is copied p times (the length of sentence A) to be extended into a matrix of size [p, s, x], denoted as a. Similarly, the above-context vector is extended into a matrix of size [p, s, x], denoted as b.

[0208] The dynamic annotation model 3103 calculates the vector distance between vector a and vector b, that is, calculates the squares of vector a and vector b and sums them up in one dimension to obtain a vector matrix of p * s, that is, obtains the emission probability of each word in sentence B to s type labels, denoted as vector c.

[0209] Finally, the dynamic annotation model 3103 normalizes vector c using the softmax function. The purpose of normalization is to make the sum of the emission probabilities of each word in sentence A to each label equal to 1.

[0210] Such as Figure 10 is a schematic diagram of the emission probability matrix after normalizing vector c. In the normalized emission probability matrix, the sum of the emission probabilities of each word in the unannotated text sequence to each annotation label is 1.

[0211] The dynamic annotation model 3103 can introduce entity features during the decoding process to increase the emission probability scores of each word in the entity to the corresponding annotation labels.

[0212] For example, the unlabeled text sequence is "Buy a plane ticket to Shenzhen today", and the slot labels of the unlabeled text sequence are "time" and "destination". The place entity corresponds to the destination slot value, and the time entity corresponds to the time slot value. The unlabeled text sequence "Buy a plane ticket to Shenzhen today" has a place entity - Shenzhen, and a time entity - tomorrow. At this time, the emission probability score of each character in the entity to the corresponding labeled label can be increased. Specifically, the emission probability score of the character "today" to the labeled label "B-time" is increased (for example, increased by 0.6), the emission probability score of the character "day" to the labeled label "I-time" is increased (for example, increased by 0.5), the emission probability score of the character "deep" to the labeled label "B-destination" is increased (for example, increased by 0.5), and the emission probability score of the character "Shenzhen" to the labeled label "I-destination" is increased (for example, increased by 0.4). The following is obtained: Figure 5 The emission probability matrix after the introduction of entity features is shown.

[0213] like Figure 11 As shown, Figure 11 Schematic diagram of the normalized emission probability matrix after the dynamic annotation model 3103 introduces entity features.

[0214] Finally, the dynamic tagging model 3103 decodes the transmission probability from each word to each tag in the unlabeled text sequence and the transition probability between the tagged tags in the text sequence with slot tags, and outputs the tagging result of the unlabeled text sequence.

[0215] Dynamic labeling model 3103 can use the Viterbi algorithm to combine the emission probabilities of each word in the unlabeled text sequence to each labeled label and the transition probabilities between labeled labels in the text sequence with slot labels to perform a Viterbi solution to obtain an optimal solution path. The optimal solution path is the highest probability sequence of the unlabeled text sequence. Here, the highest probability sequence of the unlabeled data refers to the label with the highest probability of each word in the unlabeled text sequence being assigned to each labeled label.

[0216] Exemplarily, the Viterbi algorithm may include: when the path with the highest probability from each word to each label in the data to be labeled passes through a certain point in the fence network, then the subpath from the starting point to that point must also be the most probable path from the starting point to that point; when there are k states at the i-th moment, then there are k shortest paths from the k states at the start to the i-th moment, and the final shortest path must pass through one of them.

[0217] Generate a labeling sequence based on the optimal solution path.

[0218] Exemplarily, the highest probability annotation sequence of the data to be annotated is calculated by the Viterbi algorithm. When calculating the shortest path of the (i + 1)-th state, only the shortest path from the start to the current k state values and the shortest path from the current state value to the (i + 1)-th state value need to be considered.

[0219] The unannotated text sequence is "Buy a ticket to Shenzhen today", and the output annotation sequence is ""Buy - O"" ""Today - B - Time"" ""Day - O - I - Time"" ""Go - O"" ""Shen - B - Destination"" ""Zhen - I - Destination"" ""Of - O"" ""Airplane - O"" ""Ticket - O""". The predefined slot positions are the time slot position and the destination slot position. According to the annotation sequence, the value of the time slot position can be obtained as "tomorrow", and the value of the destination slot position can be obtained as "Shenzhen".

[0220] It should be noted that when the dynamic annotation model 3103 starts to annotate the slot labels of the unannotated text sequence, the server 200 can still obtain the text sequence with slot labels. It can be understood that after an artificial annotates a text sequence with slot labels, the dynamic annotation model 3103 may have annotated 3 text sequences.

[0221] The server 200 obtains the text sequence with slot labels in real time. The more the number of text sequences with slot labels, the higher the accuracy of the dynamic annotation model 3103 in annotating the slot labels of the unannotated text sequence. Then, the more accurate the slot values of the text sequence extracted by the dynamic annotation model 3103 according to the slot labels of the text sequence.

[0222] As Figure 12 shown, Figure 12 It is a schematic diagram of the synchronous progress of the dynamic annotation model 3103 annotating the slot labels of the unannotated text sequence and the artificial annotation of the slot labels of the unannotated text sequence.

[0223] S1201. The server 200 obtains the m-th text sequence with slot labels.

[0224] The m-th text sequence with slot labels is manually annotated in real time. The electronic device 100 sends the m-th text sequence with slot labels to the server 200, and the server 200 receives and saves the m-th text sequence with slot labels.

[0225] S1202. The server 200 updates the training data of the dynamic annotation model 3103.

[0226] Before the server 200 obtains the m-th text sequence with slot labels, the server 200 has already saved m - 1 text sequences with slot labels. After the server 200 obtains the m-th text sequence with slot labels, the server 200 adds the m-th text sequence with slot labels to the training data, that is, the training data already has m text sequences with slot labels.

[0227] S1203. The server 200 annotates the (m + 1)-th, (m + 2)-th, and (m + 3)-th unannotated text sequences according to the m text sequences with slot labels through the dynamic annotation model 3103.

[0228] The number of unannotated text sequences annotated by the dynamic annotation model 3103 is closely related to the performance of the server annotation and the time for manually annotating an unannotated text sequence. Here, it is illustrated that one unannotated text sequence is manually annotated and three unannotated text sequences are annotated by the dynamic annotation model 3103.

[0229] Specifically, for how the dynamic annotation model 3103 annotates the (m + 1)-th, (m + 2)-th, and (m + 3)-th unannotated text sequences according to the text sequences with slot labels, please refer to Figure 4 the embodiment and will not be elaborated here.

[0230] S1204. The server 200 obtains the (m + 4)-th text sequence with slot labels.

[0231] The (m + 4)-th text sequence with slot labels is manually and real-time annotated. The electronic device 100 sends the (m + 4)-th text sequence with slot labels to the server 200, and the server 200 receives and saves the (m + 4)-th text sequence with slot labels.

[0232] S1205. The server 200 updates the training data of the dynamic annotation model 3103.

[0233] Before the server 200 obtains the (m + 4)-th text sequence with slot labels, the server 200 has already saved m text sequences with slot labels. After the server 200 obtains the (m + 4)-th text sequence with slot labels, the server 200 adds the (m + 4)-th text sequence with slot labels to the training data, that is, the training data already has m + 1 text sequences with slot labels.

[0234] S1206. The server 200 annotates the (m + 5)-th, (m + 6)-th, and (m + 7)-th unannotated text sequences according to the m + 1 text sequences with slot labels through the dynamic annotation model 3103. Specifically, for how the dynamic annotation model 3103 annotates the (m + 5)-th, (m + 6)-th, and (m + 7)-th unannotated text sequences according to the text sequences with slot labels, please refer toFigure 4 Examples are not elaborated here.

[0235] S1207. The server 200 obtains the (m + 8)-th text sequence with slot tags.

[0236] The (m + 8)-th text sequence with slot tags is manually and real-time annotated. The electronic device 100 sends the (m + 8)-th text sequence with slot tags to the server 200, and the server 200 receives and saves the (m + 8)-th text sequence with slot tags.

[0237] S1208. The server 200 updates the training data of the dynamic annotation model 3103.

[0238] Before the server 200 obtains the (m + 8)-th text sequence with slot tags, the server 200 has already saved m + 1 text sequences with slot tags. After the server 200 obtains the (m + 8)-th text sequence with slot tags, the server 200 adds the (m + 8)-th text sequence with slot tags to the training data, that is, the training data already has m + 2 text sequences with slot tags.

[0239] S1209. The server 200 annotates the (m + 9)-th, (m + 10)-th, and (m + 11)-th unannotated text sequences according to the m + 2 text sequences with slot tags through the dynamic annotation model 3103.

[0240] Specifically, for how the dynamic annotation model 3103 annotates the (m + 9)-th, (m + 10)-th, and (m + 11)-th unannotated text sequences according to the text sequences with slot tags, please refer to Figure 4 Examples are not elaborated here.

[0241] The server 200 can also continuously obtain the text sequences with slot tags sent by the electronic device 100, and annotate the unannotated text sequences according to the text sequences with slot tags. This application will not elaborate further here.

[0242] According to the law of large numbers, the more text sequences with slot tags the server 200 obtains, the more accurate the result of the dynamic annotation model 3103 annotating the unannotated text sequences according to the text sequences with slot tags will be.

[0243] As Figure 13 shown, Figure 13 is a flowchart of a text annotation extraction method provided by this application. The method includes:

[0244] S1301. The electronic device 100 receives a task item.

[0245] Task items are created by users. The task items can be to label slot tags (such as "time" and "destination") for unlabeled text sequences, and extract the slot values of the text sequences based on the slot tags of the text sequences.

[0246] The unlabeled text sequences are data belonging to a specific domain (such as the ticket purchase domain). The slots of the unlabeled text sequences are predefined. For example, the slots of the unlabeled text sequences can include a time slot and a destination slot. The slot values in the unlabeled text sequences can be extracted according to the annotation results of the text sequences by annotating the unlabeled text sequences. For example, the unlabeled text sequence is "Buy a ticket to Shanghai today". The annotation result of the unlabeled text sequence "Buy a ticket to Shanghai today" is ""Buy - O "Today - B - time" "day - I - time" "go - O" "Shanghai - B - destination" "hai - I - destination" "of - O" "plane - O" "ticket - O"". From the annotation result, it can be known that the slot value of the time slot is "today", and the slot value of the destination slot is "Shanghai".

[0247] S1302. If there is no automatic annotation model service, or the preset tags of the existing automatic annotation model service do not match the slot tags of the task items, the electronic device 100 receives the user selection operation.

[0248] In response to the task items created by the user, the electronic device 100 provides a selection entry for the user.

[0249] Exemplarily, as Figure 14 shown, Figure 14 an exemplary user interface for providing a selection entry for the user on the electronic device 100 is exemplarily shown.

[0250] The electronic device 100 receives the operation of the user creating task items. In response to this operation, the electronic device 100 displays a user interface 500 as Figure 5 shown. The user interface 500 includes a "First Automatic Annotation Model Service" control 5101, a "Second Automatic Annotation Model Service" control 5102, a "Nth Automatic Annotation Model Service" control 5102, and a "Dynamic Annotation Model Service 3101" control 5103.

[0251] The "First Automatic Annotation Model Service" corresponds to a first automatic annotation model. The "First Automatic Annotation Model Service" can annotate the first preset (such as "movie name") tags of the unlabeled text sequences through the first automatic annotation model. The first automatic annotation model is trained through a large number of text sequences with the first preset (such as "movie name") tags. The "First Automatic Annotation Model Service" can only be used to annotate the first preset (such as "movie name") tags of the unlabeled text sequences through the first automatic annotation model.

[0252] The "Second Automatic Annotation Model Service" corresponds to a second automatic annotation model. The "Second Automatic Annotation Model Service" can annotate the second preset (such as "cinema name") tags of an unannotated text sequence through the second automatic annotation model. The second automatic annotation model is trained with a large number of text sequences with the second preset (such as "cinema name") tags. The "Second Automatic Annotation Model Service" can only be used to annotate the second preset (such as "cinema name") tags of an unannotated text sequence through the second automatic annotation model.

[0253] It can be understood that the first preset tag and the second preset tag belong to different types of tags.

[0254] For the introduction of the "Dynamic Annotation Model Service 3101", please refer to the above embodiments, and it will not be elaborated here.

[0255] The "First Automatic Annotation Model Service" control 5101 can receive a user selection operation (such as a click). In response to the user selection operation, the electronic device 100 selects the "First Automatic Annotation Model Service"; the "Second Automatic Annotation Model Service" control 5102 can receive a user selection operation (such as a click). In response to the user selection operation, the electronic device 100 selects the function of the "Second Automatic Annotation Model Service"; the "Dynamic Annotation Model Service 3101" control 5105 can receive a user selection operation (such as a click). In response to the user selection operation, the electronic device 100 selects the "Dynamic Annotation Model Service 3101".

[0256] If the preset tag of the existing automatic annotation model service does not match the slot tag of the task item, the user can select the "Dynamic Annotation Model Service 3101". The "Cold Start Service" control 5104 can receive a user selection operation (such as a click). In response to the user selection operation, the electronic device 100 selects the "Dynamic Annotation Model Service 3101".

[0257] S1303. In response to the user selection operation, the electronic device 100 sends a request to start the dynamic annotation model service to the server 200.

[0258] S1304. The server 200 receives and responds to the request to start the dynamic annotation model service sent by the electronic device 100. The server 200 starts the dynamic annotation model service 3101.

[0259] When the server 200 starts the dynamic annotation model service 3101, that is, the dynamic annotation model service 3101 can start to annotate the slot tags (such as "time" and "destination") of the unannotated text sequence through the dynamic annotation model 3103.

[0260] S1305. The electronic device 100 obtains the text sequence with slot tags.

[0261] A text sequence is a combination of one or more sentences. For example, the text sequence can be "I want to buy a plane ticket for tomorrow to Beijing" or "Buy a train ticket for today to Shenzhen".

[0262] Each predefined slot in this text sequence corresponds to a slot value, each slot value corresponds to a slot label, and each text sequence with a slot label can include one or more slot labels.

[0263] The "BIO" tagging method can be used to tag one or more slot labels in this text sequence. Among them, "B" indicates the start of the word representing the slot label in the text sequence, "I" indicates the end of the word representing the slot label in the text sequence, and "O" indicates the characters in the text sequence that are not slot labels.

[0264] As Figure 5 shown, the text sequence with slot labels can be ""Buy - O" "tomorrow - B - time" "day - I - time" "go - O" "to - B - destination" "Beijing - I - destination" "of - O" "plane - O" "ticket - O"".

[0265] The text sequence with slot labels can be manually annotated. The electronic device 100 obtains the text sequence with slot labels that is manually annotated.

[0266] S1306. The electronic device 100 sends the text sequence with slot labels to the server 200.

[0267] S1307. The server 200 receives and saves the text sequence with slot labels sent by the electronic device 100.

[0268] S1308. The server 200 determines that the number of text sequences with slot labels is greater than n, and the server 200 updates the dynamic annotation model service 3101, where n is a positive integer greater than or equal to 1.

[0269] Among them, before the update, the dynamic annotation model service 3101 can annotate the preset labels (such as "time") of the unannotated text sequence through the dynamic annotation model 3103. The server 200 determines that the number of text sequences with slot labels has reached the threshold, and the server 200 updates the dynamic annotation model service 3101. After the update, the dynamic annotation model service 3101 can be used to annotate the slot labels (such as "time" and "destination") of the unannotated text sequence through the dynamic annotation model 3103. Among them, the label types of the preset labels and the slot labels are different.

[0270] It can be understood that steps S1305, S1306, S1307, and S1308 are carried out in real time. The electronic device 100 can continuously obtain the text sequence with slot tags and send the text sequence with slot tags to the server 200. The server 200 receives the text sequence with slot tags sent by the electronic device 100. Each time the server 200 receives a text sequence with slot tags sent by the electronic device 100, the server 200 determines whether the text sequence with slot tags is greater than n. If the text sequence with slot tags is greater than n, the server 200 updates the dynamic annotation model service 3101.

[0271] S1309. After the server 200 updates the dynamic annotation model service 3101, the dynamic annotation model 3103 starts to annotate the slot tags (such as "time" and "destination") of the unannotated text sequence and extracts the slot values of the text sequence according to the slot tags of the annotated text sequence.

[0272] For example, for the unannotated text sequence "Buy a ticket to Shanghai today", the slots of the unannotated text sequence are the time slot and the destination slot. The dynamic annotation model 3103 annotates the unannotated text sequence according to the slot tags (such as "time" and "destination"). The annotation result of the unannotated text sequence "Buy a ticket to Shanghai tomorrow" is ""Buy - O" "Tomorrow - B - time" "day - I - time" "go - O" "Shanghai - B - destination" "hai - I - destination" "of - O" "ticket - O"". From the annotation result, it can be known that the slot value of the time slot of this text sequence is "today", and the slot value of the destination slot is "Shanghai".

[0273] For the dynamic annotation model 3103 to annotate the slot tags of the unannotated text sequence and extract the slot values of the text sequence according to the slot tags of the annotated text sequence, please refer to Figure 4 the embodiments, and the details will not be elaborated here.

[0274] In the above embodiments, depending on the context, the term "when..." can be interpreted to mean "if...", or "after...", or "in response to determining...", or "in response to detecting...". Similarly, depending on the context, the phrase "when determining..." or "if detecting (the stated condition or event)" can be interpreted to mean "if determining...", or "in response to determining...", or "when detecting (the stated condition or event)", or "in response to detecting (the stated condition or event)".

[0275] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the processes or functions described in the embodiments of the present application are generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center by wire (such as coaxial cable, optical fiber, digital subscriber line) or wirelessly (such as infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that the computer can access or a data storage device such as a server or data center that includes one or more integrated available media. The available medium can be a magnetic medium (such as a floppy disk, hard disk, magnetic tape), an optical medium (such as a DVD), or a semiconductor medium (such as a solid-state drive), etc.

[0276] Those of ordinary skill in the art can understand that all or part of the processes in the above embodiments of the method can be completed by instructing relevant hardware with a computer program. The program can be stored in a computer-readable storage medium. When the program is executed, it can include the processes of the above method embodiments. The foregoing storage media include: various media such as ROM or random access memory RAM, magnetic disks, or optical discs that can store program codes.

[0277] As described above, the above embodiments are only used to illustrate the technical solutions of the present application and are not intended to limit them. Although the present application has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that they can still modify the technical solutions described in the foregoing embodiments or perform equivalent replacements for some of the technical features. These modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present application.

Claims

1. A method for extracting text annotations, characterized in that, Including: The server receives a first annotation request sent by an electronic device; In response to the first annotation request, the server activates a dynamic annotation service, where the dynamic annotation service corresponds to a first dynamic annotation model, and the first dynamic annotation model is trained with a sample text sequence with a first label; The server receives the first text sequence sent by the electronic device and the first annotation information corresponding to the first text sequence, where the first text sequence includes first text information and a second label, and the first annotation information is used to indicate the position of the text with the attribute of the second label in the first text sequence, and the first label is different from the second label; The server obtains a second text sequence; When the number of the first text sequences is greater than a first threshold, the server determines the second annotation information of the second text sequence based on the first text sequence and the first annotation information through the first dynamic annotation model, and the second annotation information is used to indicate the position of the text with the attribute of the second label in the second text sequence; The server extracts the text with the attribute of the second label in the second text sequence based on the second annotation information.

2. The method according to claim 1, characterized in that, The server determines the second annotation information of the second text sequence through the first dynamic annotation model according to the first text sequence and the first annotation information, specifically including: The server determines the transition probability between the text with the attribute of the second label and the text without the attribute of the second label in the first text sequence based on the first text sequence and the first annotation information, and the text with the attribute of the second label includes the starting word with the attribute of the second label and the continuous word with the attribute of the second label; The server determines the emission probability between each word in the second text sequence and the text with the attribute of the second label and the text without the attribute of the second label based on the first text sequence and the first annotation information; The server annotates the second annotation information of the second text sequence according to the transition probability and the emission probability.

3. The method according to claim 2, characterized in that The server determines the transition probability between the text with the attribute of the second label and the text without the attribute of the second label in the first text sequence based on the first text sequence and the first annotation information, specifically including: Based on the first text sequence and the first annotation information, the server determines the first transition count between the words without the second label attribute and the words without the second label attribute in the first text sequence, the second transition count between the words without the second label attribute and the starting word with the second label attribute in the first text sequence, the third transition count between the words without the second label attribute and the continuing word with the second label attribute in the first text sequence, the fourth transition count between the starting word with the second label attribute and the words without the second label attribute in the first text sequence, the fifth transition count between the starting word with the second label attribute and the starting word with the second label attribute in the first text sequence, the sixth transition count between the starting word with the second label attribute and the continuing word with the second label attribute in the first text sequence, the seventh transition count between the continuing word with the second label attribute and the words without the second label attribute in the first text sequence, the eighth transition count between the continuing word with the second label attribute and the starting word with the second label attribute in the first text sequence, and the ninth transition count between the continuing word with the second label attribute and the continuing word with the second label attribute in the first text sequence; The server smooths the observed values of the first transition count, the second transition count, the third transition count, the fourth transition count, the fifth transition count, the sixth transition count, the seventh transition count, the eighth transition count, and the ninth transition count according to the Laplace smoothing constant; The server determines the transition probability between the words with the second label attribute and the words without the second label attribute in the first text sequence according to the first transition count after smoothing the observed values, the second transition count after smoothing the observed values, the third transition count after smoothing the observed values, the fourth transition count after smoothing the observed values, the fifth transition count after smoothing the observed values, the sixth transition count after smoothing the observed values, the seventh transition count after smoothing the observed values, the eighth transition count after smoothing the observed values, and the ninth transition count after smoothing the observed values; 4. The method according to claim 2, wherein Based on the first text sequence and the first annotation information, the server determines the emission probability between each word in the second text sequence and the words with the second label attribute and the words without the second label attribute, specifically including: The server obtains the previous context vector according to the first text information; The server obtains the next context vector according to the second text sequence; The server determines the representation of the label according to the next context vector and the first annotation information; The server calculates the vector distance between the previous context vector and the representation of the label; The server normalizes the vector distance to determine the emission probability between each word in the second text sequence and the words with the attribute of the second label and the words without the attribute of the second label.

5. The method according to claim 2, characterized in that, The server labels the second annotation information of the second text sequence according to the transition probability and the emission probability, specifically including: The server performs Viterbi solution on the second text sequence according to the Viterbi algorithm to obtain the optimal solution path in the second text sequence; wherein, in the optimal solution path, the emission probability value between each word in the second text sequence and the words with the attribute of the second label and the words without the attribute of the second label is the largest; The server labels the second annotation information of the second text sequence based on the optimal solution path.

6. The method according to any one of claims 1-5, characterized in that, The method further includes: The server obtains a third text sequence; The server determines the third annotation information of the third text sequence through the first dynamic annotation model based on the first text sequence, the first annotation information, the second text sequence, and the second annotation information, and the third annotation information is used to indicate the position of the words with the attribute of the second label in the third text sequence in the third text sequence; The server extracts the words with the attribute of the second label in the third text sequence based on the third annotation information.

7. The method according to any one of claims 1-5, characterized in that, The type of the first annotation information includes the BIO information type.

8. The method according to any one of claims 1-5, characterized in that, The type of the first label includes any one or several of time, origin, destination, vehicle type, and the type of the second label includes any one or several of time, origin, destination, vehicle type.

9. A server, characterized in that, It includes a processor and a memory; the memory is coupled to the processor, the memory is used to store computer program code, the computer program code includes computer instructions, and the processor calls the computer instructions to enable the server to execute: Receive a first annotation request sent by an electronic device; In response to the first annotation request, start a dynamic annotation service, wherein the dynamic annotation service corresponds to a first dynamic annotation model, and the first dynamic annotation model is trained by a sample text sequence with a first label; Receive the first text sequence sent by the electronic device and the first annotation information corresponding to the first text sequence, wherein the first text sequence includes first text information and a second label, the first annotation information is used to indicate the position of the words with the attribute of the second label in the first text sequence in the first text sequence, and the first label and the second label are different; Obtain a second text sequence; When the number of the first text sequences is greater than a first threshold, determine the second annotation information of the second text sequence through the first dynamic annotation model based on the first text sequence and the first annotation information, and the second annotation information is used to indicate the position of the words with the attribute of the second label in the second text sequence in the second text sequence; Extract the text with the attributes of the second label in the second text sequence based on the second annotation information.

10. The server according to claim 9, wherein the processor is further configured to call the computer instructions to cause the server to execute: Based on the first text sequence and the first annotation information, determine the transition probability between the text with the attributes of the second label and the text without the attributes of the second label in the first text sequence, where the text with the attributes of the second label includes the starting word with the attributes of the second label and the continuing word with the attributes of the second label; Based on the first text sequence and the first annotation information, determine the emission probability between each word in the second text sequence and the text with the attributes of the second label and the text without the attributes of the second label; Annotate the second annotation information of the second text sequence according to the transition probability and the emission probability.

11. The server according to claim 10, wherein the processor is further configured to call the computer instructions to cause the server to execute: Based on the first text sequence and the first annotation information, determine the first transition count between the text without the attributes of the second label and the text without the attributes of the second label in the first text sequence, the second transition count between the text without the attributes of the second label and the starting word with the attributes of the second label in the first text sequence, the third transition count between the text without the attributes of the second label and the continuing word with the attributes of the second label in the first text sequence, the fourth transition count between the starting word with the attributes of the second label and the text without the attributes of the second label in the first text sequence, the fifth transition count between the starting word with the attributes of the second label and the starting word with the attributes of the second label in the first text sequence, the sixth transition count between the starting word with the attributes of the second label and the continuing word with the attributes of the second label in the first text sequence, the seventh transition count between the continuing word with the attributes of the second label and the text without the attributes of the second label in the first text sequence, the eighth transition count between the continuing word with the attributes of the second label and the starting word with the attributes of the second label in the first text sequence, and the ninth transition count between the continuing word with the attributes of the second label and the continuing word with the attributes of the second label in the first text sequence; Smooth the observed values of the first transition count, the second transition count, the third transition count, the fourth transition count, the fifth transition count, the sixth transition count, the seventh transition count, the eighth transition count, and the ninth transition count according to the Laplace smoothing constant; Determine the transition probability between the words with the second label attribute and the words without the second label attribute in the first text sequence according to the first transition count after desmoothing the observed values, the second transition count after desmoothing the observed values, the third transition count after desmoothing the observed values, the fourth transition count after desmoothing the observed values, the fifth transition count after desmoothing the observed values, the sixth transition count after desmoothing the observed values, the seventh transition count after desmoothing the observed values, the eighth transition count after desmoothing the observed values, and the ninth transition count after desmoothing the observed values.

12. The server according to claim 10, wherein the processor is further configured to call the computer instructions to cause the server to execute: The server obtains an upstream vector according to the first text information; The server obtains a downstream vector according to the second text sequence; The server determines the representation of the label according to the downstream vector and the first annotation information; The server calculates the vector distance between the upstream vector and the representation of the label; Normalize the vector distance to determine the emission probability between each word in the second text sequence and the words with the second label attribute and the words without the second label attribute.

13. The server according to claim 10, wherein the processor is further configured to call the computer instructions to cause the server to execute: Perform Viterbi solution on the second text sequence according to the Viterbi algorithm to obtain the optimal solution path in the second text sequence; where, In the optimal solution path, maximize the probability of the final path between each word in the second text sequence and the words with the second label attribute and the words without the second label attribute; Based on the third annotation information, extract the words with the second label attribute in the third text sequence.

14. The server according to any one of claims 9-13, wherein the processor is further configured to call the computer instructions to cause the server to execute: The type of the first annotation information includes the BIO information type.

15. The server according to any one of claims 9-13, wherein the processor is further configured to call the computer instructions to cause the server to execute: The type of the first label includes any one or several of time, origin, destination, vehicle type, and the type of the second label includes any one or several of time, origin, destination, vehicle type.

16. A computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the processor executes the method according to any one of claims 1-8.

Images