Method and device for determining case data of target domain, equipment and storage medium

Abstract

The invention relates to the technical field of data, and discloses a method and device for determining case data in a target domain, equipment and a storage medium, and the method comprises the steps: calculating the similarity between a target initial text in an initial text set and other initial texts, so as to determine a first initial text combination of which the similarity is greater than a preset similarity, the similarity of the second initial text combination is smaller than or equal to the preset similarity; wherein the target initial text is any initial text in the initial text set; determining a to-be-deleted initial text from the first initial text combination, and deleting the to-be-deleted initial text from the initial text set to obtain a use case text set; and performing similarity calculation on the use case text set and different field word lists, and taking the field to which the field word list corresponding to the highest similarity belongs as a target field of the use case text set. According to the method, the target field to which the case text set belongs can be accurately determined.

Description

Technical Field

[0001] The present application relates to the field of data technology, and in particular to a method, apparatus, device and storage medium for determining use case data in a target field. Background Art

[0002] At present, learning and testing of large models requires a large amount of adaptive use case data, especially when conducting domain-specific learning and testing of large models, a large amount of use case data belonging to the target domain is required. However, it is difficult to classify and organize the various types of data in the massive amount of data collected, and it is impossible to quickly determine the target domain to which the use case data belongs, thereby failing to enrich the use case data in the target domain. Summary of the invention

[0003] In view of the above problems, the present application provides a method, apparatus, device and storage medium for determining use case data of a target domain, which are used to quickly and accurately determine the target domain to which the use case data belongs.

[0004] According to one aspect of the present application, a determination method is provided, and the method for determining use case data of the target domain includes: calculating the similarity between the target initial text and other initial texts in the initial text set to determine a first initial text combination whose similarity is greater than a preset similarity, and a second initial text combination whose similarity is less than or equal to the preset similarity; wherein the target initial text is any initial text in the initial text set; determining the initial text to be deleted from the first initial text combination, and deleting the initial text to be deleted from the initial text set to obtain a use case text set; calculating the similarity between the use case text set and different domain vocabularies, and taking the domain to which the domain vocabularies corresponding to the highest similarity belong as the target domain of the use case text set.

[0005] In an optional manner, the calculation obtains the similarity between the target initial text and other initial texts in the initial text set to determine a first initial text combination having a similarity greater than a preset similarity and a second initial text combination having a similarity less than or equal to the preset similarity, further comprising: summing the number of words in the target initial text with the number of words in each of the other initial texts, respectively, to obtain the sum of words of multiple initial text combinations consisting of the target initial text and each of the other initial texts; determining the number of matching words in the target initial text and each of the other initial texts, and taking the quotient of the number of each matching word and the corresponding sum of words, to obtain the similarity corresponding to each initial text combination; taking the initial text combination having a similarity greater than the preset similarity as the first initial text combination, and taking the initial text combination having a similarity less than or equal to the preset similarity as the second initial text combination.

[0006] In an optional manner, determining the initial text to be deleted from the first initial text combinations further includes: determining the initial text with the highest appearance rate from all the first initial text combinations, and determining the other initial texts in all the first initial text combinations, except the initial text with the highest appearance rate, as the initial text to be deleted; or randomly selecting retained initial texts from all the first initial text combinations, and determining the other initial texts in all the first initial text combinations, except the retained initial text, as the initial text to be deleted.

[0007] In an optional manner, the similarity calculation between the use case text set and different domain vocabularies further includes: vectorizing each use case text in the use case text set to obtain a vector component corresponding to each use case text; wherein the number of the vector components is the same as the number of preset words in the target domain vocabulary, and the target domain vocabulary is a vocabulary corresponding to any domain; based on the vector components corresponding to each use case text and the word frequency vector of each preset word in the target domain vocabulary, the similarity between the use case text set and the target domain vocabulary is calculated to obtain the similarity between the use case text and the different domain vocabularies.

[0008] In an optional manner, the similarity between the use case text set and the target domain vocabulary is calculated based on the vector components corresponding to each use case text and the word frequency vectors of each preset word in the target domain vocabulary, further comprising: constructing a first vector matrix based on the vector components corresponding to each use case text, and calculating a first module length of the first vector matrix; constructing a second vector matrix based on the word frequency vectors of each preset word in the target domain vocabulary, and calculating a second module length of the second vector matrix; dividing the product of the first vector matrix and the second vector matrix by the product of the first module length and the second module length to obtain the similarity between the use case text set and the target domain vocabulary.

[0009] In an optional manner, the determination method also includes: performing word segmentation processing on the real-time text data belonging to the target domain, and preprocessing the real-time text data after the word segmentation processing to obtain real-time words belonging to the target domain; if there are target historical words that are the same as the real-time words in the target domain vocabulary, then the word frequency of the target historical words in the target domain vocabulary is increased, and the real-time words that do not exist in the target domain vocabulary are updated to the target domain vocabulary to obtain an updated target domain vocabulary.

[0010] In an optional manner, before determining the initial text to be deleted from the first initial text combination, the determination method also includes: detecting whether the first initial text combination exists; if not, using the initial text set as the use case text set; if so, executing the step of determining the initial text to be deleted from the first initial text combination, and deleting the initial text to be deleted from the initial text set to obtain the use case text set.

[0011] According to another aspect of the present application, a determination device is provided, wherein the determination device for use case data of a target domain comprises: a calculation module, configured to calculate the similarity between a target initial text and other initial texts in an initial text set, so as to determine a first initial text combination having a similarity greater than a preset similarity, and a second initial text combination having a similarity less than or equal to the preset similarity; wherein the target initial text is any initial text in the initial text set; a deletion module, configured to determine an initial text to be deleted from the first initial text combination, and delete the initial text to be deleted from the initial text set, so as to obtain a use case text set; and a determination module, configured to perform similarity calculation between the use case text set and different domain vocabularies, and to use the domain to which the domain vocabularies corresponding to the highest similarity belong as the target domain of the use case text set.

[0012] According to one aspect of the present application, an electronic device is provided, including: a controller; and a memory for storing one or more programs, wherein when the one or more programs are executed by the controller, the above-mentioned determination method is executed.

[0013] According to one aspect of the present application, a computer-readable storage medium is also provided, on which computer-readable instructions are stored. When the computer-readable instructions are executed by a processor of a computer, the computer executes the above-mentioned determination method.

[0014] According to one aspect of the present application, a computer program product or a computer program is also provided, the computer program product or the computer program comprising computer instructions, the computer instructions being stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes the above-mentioned determination method.

[0015] The present application calculates the similarity between each pair of initial texts in the initial text set to determine a first initial text combination whose similarity is greater than a preset similarity, and determines the initial text to be deleted from the first initial text combination to avoid initial texts with similarity increasing the overlap of the use case text set, thereby making the calculated use case text set more accurate in similarity with different field vocabularies, and more accurately determining the target field to which the use case text set belongs.

[0016] The above description is only an overview of the technical solution of the embodiment of the present application. In order to more clearly understand the technical means of the present application, it can be implemented in accordance with the contents of the specification. In order to make the above and other purposes, features and advantages of the present application more obvious and easy to understand, the specific implementation methods of the present application are listed below. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] The drawings herein are incorporated into the specification and constitute a part of the specification, illustrate embodiments consistent with the present application, and are used together with the specification to explain the principles of the present application. Obviously, the drawings described below are only some embodiments of the present application, and for ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0018] Figure 1 It is a flowchart of a method for determining use case data in a target field shown in an exemplary embodiment of the present application.

[0019] Figure 2 is based on Figure 1 The exemplary embodiment shown is a flowchart of another method for determining use case data in a target domain.

[0020] Figure 3 is based on Figure 1 The exemplary embodiment shown is a flowchart of another method for determining use case data in a target domain.

[0021] Figure 4 is based on Figure 1 The exemplary embodiment shown is a flowchart of another method for determining use case data in a target domain.

[0022] Figure 5 It is a structural diagram of a device for determining use case data in a target domain shown in an exemplary embodiment of the present application.

[0023] Figure 6 It is a structural diagram of a computer system of an electronic device shown in an exemplary embodiment of the present application. DETAILED DESCRIPTION

[0024] Here, exemplary embodiments will be described in detail, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present application. Instead, they are only examples of devices and methods consistent with some aspects of the present application as detailed in the attached claims.

[0025] The block diagrams shown in the accompanying drawings are merely functional entities and do not necessarily correspond to physically independent entities. That is, these functional entities may be implemented in software form, or in one or more hardware modules or integrated circuits, or in different networks and / or processor devices and / or microcontroller devices.

[0026] The flowcharts shown in the accompanying drawings are only exemplary and do not necessarily include all the contents and operations / steps, nor must they be executed in the order described. For example, some operations / steps can be decomposed, and some operations / steps can be combined or partially combined, so the actual execution order may change according to actual conditions.

[0027] The term "multiple" as used in this application refers to two or more than two. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. The character " / " generally indicates that the related objects are in an "or" relationship.

[0028] The classification and management of existing use case data is rather chaotic, especially for the use case data required for large models. The massive use case data cannot be accurately divided into fields, making it impossible to enrich the use case data in the target field.

[0029] To this end, one aspect of the present application provides a method for determining use case data in a target domain. Figure 1 , Figure 1 1 is a flow chart of a method for determining use case data in a target domain according to an exemplary embodiment of the present application. The determination method at least includes S110 to S130, which are described in detail as follows:

[0030] S110: Calculate the similarity between the target initial text and other initial texts in the initial text set to determine a first initial text combination with a similarity greater than a preset similarity and a second initial text combination with a similarity less than or equal to the preset similarity; wherein the target initial text is any initial text in the initial text set.

[0031] Exemplarily, each initial text in the initial text set is traversed, and the traversed initial text is used as the target historical initial text; the target initial text is calculated for similarity with other initial texts in the initial text set one by one, and the similarity corresponding to multiple initial text combinations including the target initial text is obtained. Similarly, the similarity is calculated for each pair of initial texts in the initial text set to obtain the similarity corresponding to multiple initial text combinations. It is worth noting that the similarity of existing initial text combinations will not be repeated during the calculation process. For example, if the similarity between the first initial text and the second initial text has been calculated, the similarity between the second initial text and the first initial text will not be calculated again.

[0032] This embodiment does not limit the specific value of the preset similarity, which is only used as a parameter to distinguish the first initial text combination from the second initial text combination. The similarity of the two initial texts in the initial text combination (i.e., the first initial text combination) greater than the preset similarity is high, and the initial texts with high similarity can be deduplicated to ensure the uniqueness of the use case text. The similarity of the two initial texts in the second initial text combination of this embodiment is low.

[0033] The initial text is the text obtained after word segmentation based on the cleaned data set, including but not limited to nouns, verbs, adverbs, etc. For example: the collected initial data set is cleaned to remove invisible characters, normalize spaces, etc., and then the cleaned data set is word segmented to obtain the corresponding initial text set.

[0034] At present, the data format is generally standardized and cleaned without considering the factors of the data itself. In some embodiments, in order to protect personal privacy information, sensitive information needs to be replaced during the data cleaning process, for example: Email replacement: email address is replaced with placeholder [EMAIL]; IP address replacement: including IPv4 or IPv6 address replaced with [IPADDRESS]; ID number replacement: ID number is replaced with placeholder [IDCARD]; phone number replacement: number including international phone number format, landline format, mobile phone number format, replaced with designated placeholder [PHONE], special service number format (such as 110, 119) is not replaced; bank card number replacement: credit card and debit card number, replaced with designated placeholder [BANKCARD]. By replacing sensitive data, the cleaned data set is made more secure.

[0035] In some embodiments, in order to improve the accuracy of text similarity, the initial text needs to be preprocessed to eliminate the influence of words with low correlation on the similarity, for example, stop words, prepositions and other words that do not affect the substantive meaning of the initial text are removed.

[0036] S120: Determine the initial text to be deleted from the first initial text combination, and delete the initial text to be deleted from the initial text set to obtain the use case text set.

[0037] Each first initial text combination includes two initial texts, and the initial texts in all the first initial texts are analyzed to select initial texts with the same similarity as initial texts to be deleted, so as to perform deduplication processing on the initial text set to obtain a use case text set.

[0038] Exemplarily, the initial text with the highest appearance rate is determined from all first initial text combinations, and the other initial texts except the initial text with the highest appearance rate in all first initial text combinations are determined as the initial texts to be deleted.

[0039] That is, the occurrence rate of the initial texts in all the first initial text combinations is counted to retain the initial text with the highest occurrence rate, and the other initial texts in all the first initial text combinations are deleted. For example, if the initial texts in the first initial text are initial texts 1, 2, and 3, among which initial text 2 has the highest occurrence rate, then initial text 2 is retained, and initial texts 1 and 3 are used as initial texts to be deleted.

[0040] In another exemplary embodiment, a retained initial text is randomly selected from all first initial text combinations, and other initial texts except the retained initial text in all first initial text combinations are determined as initial texts to be deleted.

[0041] That is, one or more initial texts are randomly selected from all the first initial text combinations to be retained, and other initial texts in the first initial text combination are deleted. For example, the initial texts in the first initial text are initial texts 1, 2, and 3, and initial text 2 is randomly selected to be retained, and initial texts 1 and 3 are used as initial texts to be deleted.

[0042] S130: Calculate similarity between the use case text set and different domain vocabularies, and use the domain to which the domain vocabularies with the highest similarity belong as the target domain of the use case text set.

[0043] Each domain vocabulary has the characteristics of the domain in which it is located. The words in the vocabulary are suitable for the domain in which it is located, and the vocabulary includes common words, common nouns, etc. in the domain.

[0044] By calculating the similarity between the use case text set and all domain vocabularies one by one, the similarity between the use case text and each domain vocabularies can be calculated, and the domain with the highest similarity can be selected as the target domain to which the use case text set belongs.

[0045] In some embodiments, if there are multiple fields corresponding to the calculated highest similarity, it indicates that the use case text set may belong to multiple fields.

[0046] This embodiment calculates the similarity between each pair of initial texts in the initial text set to determine a first initial text combination whose similarity is greater than a preset similarity, and determines the initial text to be deleted from the first initial text combination to avoid initial texts with similarity increasing the overlap of the use case text set, thereby making the calculated use case text set more accurate in similarity with different domain vocabularies, and more accurately determining the target domain to which the use case text set belongs.

[0047] In another exemplary embodiment of the present application, how to determine the first initial text combination and the second initial text combination is described in detail. Figure 2 , Figure 2 is based on Figure 1 The exemplary embodiment shown is a flow chart of another method for determining use case data in a target domain. Figure 1 The S110 shown further includes S210 to S230, which are described in detail as follows:

[0048] S210: Sum the number of words in the target initial text and the number of words in each other initial text to obtain the sum of words in multiple initial text combinations consisting of the target initial text and each other initial text.

[0049] The number of words in each pair of initial texts is recorded, and the sum of the words in each pair of initial texts is calculated to be the total number of samples for similarity calculation.

[0050] S220: Determine the number of matching words in the target initial text and each other initial text, and calculate the quotient of the number of each matching word and the corresponding sum of words to obtain the similarity corresponding to each initial text combination.

[0051] Matching words are identical words, that is, words that exist in both initial texts. This embodiment uses simple mathematical operations to perform a simple quotient operation on the number of matching words and the total number of words, and can quickly obtain a quotient value, so that the quotient value is used as the similarity between the initial texts, that is, the similarity corresponding to the corresponding initial text combination.

[0052] S230: taking an initial text combination whose similarity is greater than a preset similarity as a first initial text combination, and taking an initial text combination whose similarity is less than or equal to the preset similarity as a second initial text combination.

[0053] This embodiment is introduced as an example. The target initial text is initial text 1, and the similarity between initial text 1 and each other initial text is obtained. Specifically:

[0054] Initial text 1: XX model is luxurious with six seats and large space, making driving and traveling more comfortable

[0055] Initial text 2: XX model Hongmeng intelligent car system enjoys a new technological driving experience

[0056] Initial text 3: XX model is spacious and luxurious with six seats and large space for driving, making every trip unforgettable

[0057] Initial text 4: XX model is a must-have for driving and traveling. The large space meets all needs.

[0058] Initial text 5: XX model is comfortable and luxurious with six seats and large space, making driving more comfortable

[0059] Similarity between initial text 1 and initial text 2:

[0060] (J(1,2)=\frac{|{XX model, driving}|}{|{XX model, luxury, six seats, large, space, driving, travel, more, comfortable, Hongmeng, smart, car, system, enjoy, technology, driving, new, experience}|}=\frac{1}{17}), that is, the similarity between initial text 1 and initial text 2 is 1 / 17.

[0061] Similarity between initial text 1 and initial text 3:

[0062] (J(1,3)=\frac{|{XX model, six seats, space, driving, travel}|}{|{XX model, luxury, six seats, extra large, space, driving, travel, more, comfortable, spacious, luxurious, large, makes, every, trip, unforgettable, memory}|}=\frac{5}{17}), that is, the similarity between initial text 1 and initial text 3 is 5 / 17.

[0063] Similarity between initial text 1 and initial text 4:

[0064] (J(1,4)=\frac{|{XX model, extra large, space, driving}|}{|{XX model, luxury, six seats, extra large, space, driving, travel, more, comfortable, travel, essential, artifact, meet, all, needs}|}=\frac{4}{15}, that is, the similarity between initial text 1 and initial text 4 is 4 / 15.

[0065] Similarity between initial text 1 and initial text 5:

[0066] (J(1,5)=\frac{|{XX model, six seats, large space, driving, travel, comfortable}|}{|{XX model, luxury, six seats, large space, driving, travel, more, comfortable, luxurious, making, more, warm}|}=\frac{7}{13}, that is, the similarity between initial text 1 and initial text 5 is 7 / 13.

[0067] The target initial text is initial text 2, so as to obtain the similarity between initial text 2 and each other initial text, and the similarity between initial text 1 and 2 is not calculated repeatedly.

[0068] Similarity between initial text 2 and initial text 3:

[0069] (J(2,3)=\frac{|{XX model, driving}|}{|{XX model, Hongmeng, intelligent, car, system, enjoy, technology, driving, new, experience, spacious, luxurious, six seats, large, space, travel, makes, every, trip, unforgettable, memory}|}=\frac{2}{21}=\frac{1}{5}), that is, the similarity between initial text 2 and initial text 3 is 1 / 5.

[0070] Similarity between initial text 2 and initial text 4:

[0071] (J(2,4)=\frac{|{XX model, driving}|}{|{XX model, Hongmeng, intelligent, car machine, system, enjoy, technology, driving, new, experience, travel, essential, artifact, large, space, meet, all, needs}|}=\frac{3}{18}=\frac{1}{6}), that is, the similarity between initial text 2 and initial text 4 is 1 / 6.

[0072] Similarity between initial text 2 and initial text 5:

[0073] (J(2,5)=\frac{|{XX model, driving}|}{|{XX model, Hongmeng, intelligent, car, system, enjoy, technology, driving, new, experience, comfortable, luxurious, six seats, large, space, makes, travel, more, warm}|}=\frac{2}{19}), that is, the similarity between initial text 2 and initial text 5 is 2 / 19.

[0074] The target initial text is initial text 3, so as to obtain the similarity between initial text 3 and each other initial text, and the similarity between initial text 1 and 3, and initial text 2 and 3 is not calculated repeatedly.

[0075] Similarity between initial text 3 and initial text 4:

[0076] (J(3,4)=\frac{|{XX model, driving, space}|}{|{XX model, spacious, luxurious, six seats, large, space, driving, travel, makes, every, trip, unforgettable, memory, outing, essential, artifact, super large, meets, all, needs}|}=\frac{3}{20}), that is, the similarity between initial text 3 and initial text 4 is 3 / 20.

[0077] Similarity between initial text 3 and initial text 5:

[0078] (J(3,5)=\frac{|{XX model, luxurious, six seats, space, driving, traveling}|}{|{XX model, spacious, luxurious, six seats, large, space, driving, traveling, makes, every, trip, unforgettable, memories, comfortable, extra-large, makes, more, warm}|}=\frac{6}{18}=\frac{1}{3}), that is, the similarity between initial text 3 and initial text 5 is 1 / 3.

[0079] The target initial text is initial text 4, in order to obtain the similarity between initial text 4 and each other initial text, and the similarity between initial text 4 and initial text 5:

[0080] (J(4,5)=\frac{|{XX model, driving, extra-large, space}|}{|{XX model, driving, travel, essential, artifact, extra-large, space, meets, all, needs, comfortable, luxurious, six-seater, makes, travel, more, warm}|}=\frac{4}{17}), that is, the similarity between initial text 4 and initial text 5 is 4 / 17.

[0081] If the preset similarity is 0.3, the initial text combination greater than 0.3 is taken as the first initial text combination, the bolded object is the first initial text combination, and the others are the second initial text combination:

[0082] J(1,2)=\frac{1}{17}≈0.05

[0083] J(1,3)=\frac{5}{17}≈0.29

[0084] J(1,4)=\frac{4}{15}≈0.24

[0085] J(1,5)=\frac{7}{13}≈0.41

[0086] J(2,3)=\frac{1}{5}≈0.2

[0087] J(2,4)=\frac{1}{6}≈0.17

[0088] J(2,5)=\frac{2}{19}≈0.11

[0089] J(3,4)=\frac{3}{20}≈0.15

[0090] J(3,5)=\frac{1}{3}≈0.33

[0091] J(4,5)=\frac{4}{17}≈0.24

[0092] In another exemplary embodiment of the present application, how to calculate the similarity between the use case text set and the vocabulary of different fields is described in detail. Figure 3 , Figure 3 is based on Figure 1 The exemplary embodiment shown is a flow chart of another method for determining use case data in a target domain. Figure 1 The S130 shown further includes S310 to S320, which are described in detail as follows:

[0093] S310: vectorize each use case text in the use case text set to obtain a vector component corresponding to each use case text; wherein the number of vector components is the same as the number of preset words in the target domain vocabulary, and the target domain vocabulary is a vocabulary corresponding to any domain.

[0094] The use case text is the initial text after deduplication. During the vectorization process, it needs to be processed according to the frequency of the corresponding field. The number of vector components of the use case text is the same as the number of preset words in the corresponding field vocabulary. The value of the specific vector component can be determined according to the matching result of the use case text and the preset words. For example: if a certain use case text successfully matches a preset word, the vector component of the use case text is 1; if a certain use case text fails to match all the preset words, the vector component of the use case text is 0.

[0095] S320: Based on the vector components corresponding to each use case text and the word frequency vectors of each preset word in the target domain vocabulary, the similarity between the use case text set and the target domain vocabulary is calculated to obtain the similarity between the use case text and the vocabulary in different domains.

[0096] Through different vector similarity calculation methods, the similarity between the vector component and the word frequency vector is calculated to obtain the similarity between the use case text set and the target domain vocabulary, which is used to characterize the similarity between the use case text and the vocabulary of different fields.

[0097] Exemplarily, a first vector matrix is ​​constructed based on the vector components corresponding to each use case text, and the first modulus of the first vector matrix is ​​calculated; a second vector matrix is ​​constructed based on the frequency vectors of each preset word in the target domain vocabulary, and the second modulus of the second vector matrix is ​​calculated; the product of the first vector matrix and the second vector matrix is ​​divided by the product of the first modulus and the second modulus to obtain the similarity between the use case text set and the target domain vocabulary. The specific introduction is as follows:

[0098] The use case text is: XX model is luxurious with six seats and large space, making driving more comfortable

[0099] The target domain vocabulary is: How to drive a car, what insurance is provided, and what is required to claim when driving.

[0100] Based on the vehicle domain vocabulary, the word frequency components and vector components of the vehicle domain are determined as shown in Table 1.

[0101] Table 1: Relationship between vehicle domain word frequency components and use case text vector components

[0102] Preset words Word frequency vector Vector Components car 5 0 how 5 0 how 4 0 drive 4 1 Insurance 3 0 yes 3 0 Which 3 0 What 2 0 have 2 0 Driving 2 0 conduct 2 0 effect 2 0 hour 2 0 Claims 2 0

[0103] This embodiment determines the words that match the use case text from the vehicle field vocabulary, and the vector component corresponding to the matched words is 1, and the vector component of the unmatched words is 0, thereby constructing a first vector matrix A=[0,0,0,1,0,0,0,0,0,0,0,0,0,0]; according to the word frequency of each preset word, the word frequency vector is determined, thereby constructing a second vector matrix B=[5,5,4,4,3,3,3,2,2,2,2,2,2,2].

[0104] The similarity between the first vector and the second vector, that is, the similarity between the use case text set and the target domain vocabulary, is calculated according to the following formula.

[0105]

[0106] Among them, S represents similarity, A represents the first vector, and B represents the second vector; ||A|| represents the first modulus length of the first vector matrix, and ||B|| represents the second modulus length of the second vector matrix.

[0107] Further,

[0108] From this, we can quickly calculate S≈0.315, that is, the similarity between the use case text set and the vehicle domain vocabulary is approximately equal to 0.315. Similarly, we calculate the similarity between the use case text set and other domain vocabulary, and take the domain with the highest similarity as the target domain to which the use case text set belongs.

[0109] In another exemplary embodiment of the present application, a target domain vocabulary update process is introduced, and the specific steps include: performing word segmentation processing on the real-time text data belonging to the target domain, and preprocessing the real-time text data after the word segmentation processing to obtain real-time words belonging to the target domain; if there are target historical words that are the same as the real-time words in the target domain vocabulary, the word frequency of the target historical words in the target domain vocabulary is increased, and the real-time words that do not exist in the target domain vocabulary are updated to the target domain vocabulary to obtain an updated target domain vocabulary.

[0110] Real-time text data is unprocessed data. After word segmentation, multiple real-time words are obtained. In some embodiments, the real-time words can be further screened to exclude prepositions with weaker relevance, so that the screened real-time words have substantive meaning, and the meaning represented by them is more substantial and more in line with related fields.

[0111] If there is a target historical word identical to the real-time word in the target domain vocabulary, the real-time word is characterized as a historical word, and its frequency only needs to be increased to respond to the successful match, for example, the frequency of the historical word is increased by one.

[0112] If a real-time word does not exist in the target domain vocabulary, it is used as a new preset word and updated into the target domain vocabulary to complete the update of the target domain vocabulary.

[0113] This embodiment updates the target domain vocabulary in real time to ensure that the target domain vocabulary includes the latest real-time words, so as to avoid missing the latest real-time words and thus failing to cover the similarity of the latest real-time words, thereby ensuring the accuracy of similarity calculation.

[0114] In another exemplary embodiment of the present application, the pre-step of S120 is described in detail. Figure 4 , Figure 4 is based on Figure 1 The exemplary embodiment shown is a flow chart of another method for determining use case data in a target domain. Figure 1 Before S120 shown, it further includes S410 to S420, which are described in detail as follows:

[0115] S410: Detect whether a first initial text combination exists.

[0116] S420: If it does not exist, the initial text set is used as the use case text set.

[0117] This implementation provides different methods for determining use case text sets by detecting whether there is a first initial text combination in the initial text set. If so, the step of determining the initial text to be deleted from the first initial text combination and deleting the initial text to be deleted from the initial text set to obtain the use case text set is performed, that is, S120 is executed; if not, it indicates that the similarity between the initial texts in the initial text set is not high, and there is no need to deduplicate the initial text set, and the initial text set can be directly used as the use case text set.

[0118] Another aspect of the present application also provides a device for determining use case data in a target domain, such as Figure 5 As shown, Figure 5 It is a structural diagram of a device for determining use case data of a target domain shown in an exemplary embodiment of the present application. The determination device 500 includes: a calculation module 510, which is used to calculate the similarity between the target initial text and other initial texts in the initial text set, so as to determine a first initial text combination with a similarity greater than a preset similarity, and a second initial text combination with a similarity less than or equal to a preset similarity; wherein the target initial text is any initial text in the initial text set; a deletion module 530, which is used to determine the initial text to be deleted from the first initial text combination, and delete the initial text to be deleted from the initial text set to obtain a use case text set; a determination module 550, which is used to calculate the similarity between the use case text set and different domain vocabularies, and use the domain to which the domain vocabularies corresponding to the highest similarity belong as the target domain of the use case text set.

[0119] In another exemplary embodiment, the calculation module 510 further includes: a first calculation unit, used to sum the number of words in the target initial text with the number of words in each other initial text, respectively, to obtain the sum of words in multiple initial text combinations consisting of the target initial text and each other initial text; a second calculation unit, used to determine the number of matching words in the target initial text and each other initial text, respectively, and to calculate the quotient of the number of each matching word and the corresponding sum of words, to obtain the similarity corresponding to each initial text combination; a combination determination unit, used to take the initial text combination whose similarity is greater than a preset similarity as the first initial text combination, and take the initial text combination whose similarity is less than or equal to the preset similarity as the second initial text combination.

[0120] In another exemplary embodiment, the deletion module 530 further includes: a first deletion unit, used to determine the initial text with the highest appearance rate from all first initial text combinations, and determine the other initial texts in all first initial text combinations, except the initial text with the highest appearance rate, as the initial text to be deleted; or a second deletion unit, used to randomly select the retained initial text from all first initial text combinations, and determine the other initial texts in all first initial text combinations, except the retained initial text, as the initial text to be deleted.

[0121] In another exemplary embodiment, the determination module 550 further includes: a vector component determination unit, which is used to vectorize each use case text in the use case text set to obtain a vector component corresponding to each use case text; wherein the number of vector components is the same as the number of preset words in the target domain vocabulary, and the target domain vocabulary is a vocabulary corresponding to any domain; a vector calculation unit, which is used to calculate the similarity between the use case text set and the target domain vocabulary based on the vector components corresponding to each use case text and the word frequency vector of each preset word in the target domain vocabulary, so as to obtain the similarity between the use case text and the vocabulary of different domains.

[0122] In another exemplary embodiment, the vector calculation unit further includes: a first module length calculation module, which is used to construct a first vector matrix according to the vector components corresponding to each use case text, and calculate the first module length of the first vector matrix; a second module length calculation module, which is used to construct a second vector matrix according to the frequency vectors of each preset word in the target domain vocabulary, and calculate the second module length of the second vector matrix; a similarity calculation module, which is used to divide the product of the first vector matrix and the second vector matrix by the product of the first module length and the second module length to obtain the similarity between the use case text set and the target domain vocabulary.

[0123] In another exemplary embodiment, the determination device 500 also includes: a preprocessing module, which is used to perform word segmentation on the real-time text data belonging to the target domain, and preprocess the real-time text data after the word segmentation to obtain real-time words belonging to the target domain; an updating module, which is used to increase the word frequency of the target historical words in the target domain vocabulary if there are target historical words that are the same as the real-time words in the target domain vocabulary, and update the real-time words that do not exist in the target domain vocabulary to the target domain vocabulary to obtain an updated target domain vocabulary.

[0124] In another exemplary embodiment, the determination device 500 also includes: a detection module, used to detect whether a first initial text combination exists; a first execution module, used to use the initial text set as a use case text set if it does not exist; and a second execution module, used to determine the initial text to be deleted from the first initial text combination and delete the initial text to be deleted from the initial text set to obtain the use case text set if it exists.

[0125] The determination device of the present application calculates the similarity between each pair of initial texts in the initial text set to determine a first initial text combination whose similarity is greater than a preset similarity, and determines the initial text to be deleted from the first initial text combination to avoid initial texts with similarity increasing the overlap of the use case text set, thereby making the calculated use case text set more accurate in similarity with different domain vocabularies, and more accurately determining the target domain to which the use case text set belongs.

[0126] It should be noted that the determination device provided in the above embodiment and the determination method provided in the above embodiment belong to the same concept, wherein the specific manner in which each module and unit performs the operation has been described in detail in the method embodiment and will not be repeated here.

[0127] Another aspect of the present application provides an electronic device, including: a controller; and a memory for storing one or more programs, which, when executed by the controller, executes the above-mentioned determination method.

[0128] See also Figure 6 , Figure 6 1 is a schematic diagram of the structure of a computer system of an electronic device shown in an exemplary embodiment of the present application, which shows a schematic diagram of the structure of a computer system of an electronic device suitable for implementing an embodiment of the present application.

[0129] It should be noted that Figure 6 The computer system 600 of the electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0130] like Figure 6As shown, the computer system 600 includes a central processing unit (CPU) 601, which can perform various appropriate actions and processes according to the program stored in the read-only memory (ROM) 602 or the program loaded from the storage part 608 to the random access memory (RAM) 603, such as executing the method in the above embodiment. In the RAM 603, various programs and data required for system operation are also stored. The CPU 601, ROM 602 and RAM 603 are connected to each other through a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.

[0131] The following components are connected to the I / O interface 605: an input section 606 including a keyboard, a mouse, etc.; an output section 607 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 608 including a hard disk, etc.; and a communication section 609 including a network interface card such as a LAN (Local Area Network) card, a modem, etc. The communication section 609 performs communication processing via a network such as the Internet. A drive 610 is also connected to the I / O interface 605 as needed. A removable medium 611, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 610 as needed so that a computer program read therefrom is installed into the storage section 608 as needed.

[0132] In particular, according to an embodiment of the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present application includes a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a computer program for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication section 609, and / or installed from a removable medium 611. When the computer program is executed by a central processing unit (CPU) 601, various functions defined in the system of the present application are executed.

[0133] It should be noted that the computer-readable medium shown in the embodiment of the present application can be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium can be, for example, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media can include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM), a flash memory, an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present application, a computer-readable storage medium can be any tangible medium containing or storing a program, which can be used by an instruction execution system, device or device or used in combination with it. In the present application, a computer-readable signal medium can include a data signal propagated in a baseband or as a part of a carrier wave, wherein a computer-readable computer program is carried. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Computer-readable signal media may also be any computer-readable medium other than computer-readable storage media, which may send, propagate, or transmit programs for use by or in conjunction with an instruction execution system, apparatus, or device. The computer program contained on the computer-readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, wired, etc., or any suitable combination of the above.

[0134] The flowchart and block diagram in the accompanying drawings illustrate the possible architecture, functions and operations of the system, method and computer program product according to various embodiments of the present application. Wherein, each box in the flowchart or block diagram can represent a module, a program segment, or a part of the code, and the above-mentioned module, program segment, or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order than the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flowchart, and the combination of boxes in the block diagram or flowchart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0135] The units involved in the embodiments described in this application may be implemented by software or hardware, and the units described may also be set in a processor. The names of these units do not, in some cases, constitute limitations on the units themselves.

[0136] Another aspect of the present application further provides a computer-readable storage medium on which a computer program is stored, and when the computer program is executed by a processor, the above determination method is implemented. The computer-readable storage medium may be included in the electronic device described in the above embodiment, or may exist independently without being assembled into the electronic device.

[0137] Another aspect of the present application also provides a computer program product or a computer program, which includes a computer instruction stored in a computer-readable storage medium. A processor of a computer device reads the computer instruction from the computer-readable storage medium, and the processor executes the computer instruction, so that the computer device executes the determination method provided in each of the above embodiments.

[0138] According to one aspect of an embodiment of the present application, a computer system is also provided, including a central processing unit (CPU), which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) or a program loaded from a storage portion into a random access memory (RAM), such as executing the method in the above embodiment. In RAM, various programs and data required for system operation are also stored. CPU, ROM and RAM are connected to each other via a bus. An input / output (I / O) interface is also connected to the bus.

[0139] The following components are connected to the I / O interface: an input part including a keyboard, a mouse, etc.; an output part including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker; a storage part including a hard disk, etc.; and a communication part including a network interface card such as a LAN (Local Area Network) card, a modem, etc. The communication part performs communication processing via a network such as the Internet. A drive is also connected to the I / O interface as needed. Removable media, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., are installed on the drive as needed so that the computer program read therefrom is installed into the storage part as needed.

[0140] The above content is only a preferred exemplary embodiment of the present application and is not intended to limit the implementation scheme of the present application. A person skilled in the art can easily make corresponding changes or modifications based on the main concept and spirit of the present application. Therefore, the scope of protection of the present application shall be based on the scope of protection required by the claims.

Claims

1. A method for determining use case data in a target domain, characterized in that: The determination method comprises: Calculate the similarity between the target initial text and other initial texts in the initial text set to determine a first initial text combination whose similarity is greater than a preset similarity and a second initial text combination whose similarity is less than or equal to the preset similarity; wherein the target initial text is any initial text in the initial text set; Determining the initial text to be deleted from the first initial text combination, and deleting the initial text to be deleted from the initial text set to obtain a use case text set; The similarity between the use case text set and different domain vocabularies is calculated, and the domain to which the domain vocabularies corresponding to the highest similarity belong is taken as the target domain of the use case text set.

2. The determination method according to claim 1, characterized in that: The calculating and obtaining the similarity between the target initial text and other initial texts in the initial text set to determine a first initial text combination having a similarity greater than a preset similarity and a second initial text combination having a similarity less than or equal to the preset similarity further includes: Sum the number of words in the target initial text with the number of words in each other initial text, to obtain the sum of words in multiple initial text combinations consisting of the target initial text and each other initial text; Determine the number of matching words in the target initial text and each other initial text, and perform a quotient operation on the number of each matching word and the corresponding sum of words to obtain the similarity corresponding to each initial text combination; An initial text combination whose similarity is greater than the preset similarity is taken as a first initial text combination, and an initial text combination whose similarity is less than or equal to the preset similarity is taken as a second initial text combination.

3. The determination method according to claim 1, characterized in that: The step of determining the initial text to be deleted from the first initial text combination further comprises: Determine the initial text with the highest appearance rate from all first initial text combinations, and determine the other initial texts in all first initial text combinations except the initial text with the highest appearance rate as the initial texts to be deleted; or A retained initial text is randomly selected from all the first initial text combinations, and other initial texts except the retained initial text in all the first initial text combinations are determined as the initial texts to be deleted.

4. The determination method according to claim 1, characterized in that: The calculating of similarity between the use case text set and different domain vocabulary further includes: Vectorizing each use case text in the use case text set to obtain a vector component corresponding to each use case text; wherein the number of the vector components is the same as the number of preset words in the target domain vocabulary, and the target domain vocabulary is a vocabulary corresponding to any domain; Based on the vector components corresponding to each use case text and the word frequency vectors of each preset word in the target domain vocabulary, the similarity between the use case text set and the target domain vocabulary is calculated to obtain the similarity between the use case text and the vocabulary in different domains.

5. The determination method according to claim 4, characterized in that: The calculating the similarity between the use case text set and the target domain vocabulary based on the vector components corresponding to each use case text and the word frequency vector of each preset word in the target domain vocabulary further includes: Constructing a first vector matrix according to vector components corresponding to each use case text, and calculating a first modulus of the first vector matrix; Constructing a second vector matrix according to the word frequency vectors of each preset word in the target domain vocabulary, and calculating a second modulus of the second vector matrix; The product of the first vector matrix and the second vector matrix is ​​divided by the product of the first modulus and the second modulus to obtain the similarity between the use case text set and the target domain vocabulary.

6. The determination method according to claim 4, characterized in that: The determination method further comprises: Performing word segmentation processing on the real-time text data belonging to the target domain, and preprocessing the real-time text data after the word segmentation processing to obtain real-time words belonging to the target domain; If there is a target historical word identical to the real-time word in the target domain vocabulary, the frequency of the target historical word in the target domain vocabulary is increased, and the real-time word that does not exist in the target domain vocabulary is updated to the target domain vocabulary to obtain an updated target domain vocabulary.

7. The determination method according to any one of claims 1 to 6, characterized in that: Before determining the initial text to be deleted from the first initial text combination, the determination method further includes: Detecting whether the first initial text combination exists; If it does not exist, the initial text set is used as the use case text set; If it exists, the step of determining the initial text to be deleted from the first initial text combination and deleting the initial text to be deleted from the initial text set to obtain the use case text set is performed.

8. A device for determining use case data of a target domain, characterized in that: The determining device comprises: A calculation module, used for calculating the similarity between the target initial text and other initial texts in the initial text set, so as to determine a first initial text combination whose similarity is greater than a preset similarity and a second initial text combination whose similarity is less than or equal to the preset similarity; wherein the target initial text is any initial text in the initial text set; A deletion module, used for determining an initial text to be deleted from the first initial text combination, and deleting the initial text to be deleted from the initial text set to obtain a use case text set; The determination module is used to calculate the similarity between the use case text set and different domain vocabularies, and to use the domain to which the domain vocabularies corresponding to the highest similarity belong as the target domain of the use case text set.

9. An electronic device, characterized in that: include: Controller; A memory for storing one or more programs, which, when executed by a controller, enables the controller to implement the determination method described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: Computer-readable instructions are stored thereon, and when the computer-readable instructions are executed by a processor of a computer, the computer is caused to execute the determination method according to any one of claims 1 to 7.

Images