Data processing method, apparatus, device, and medium

Abstract

The present disclosure provides a data processing method and device, equipment and medium, computer technology field, especially in the field of intelligent education, artificial intelligence, big data technology. The implementation scheme is: obtaining a plurality of first text data; respectively encoding each first text data to determine a plurality of signature values of each first text data; for each first text data, copying the first text data into multiple parts corresponding to the plurality of signature values respectively to obtain a plurality of second text data; based on the plurality of second text data and the signature value corresponding to each second text data, obtaining a plurality of data sets; for each data set containing a first search request text, determining the first recognition result of each first search request text in the data set; and determining the second recognition result of each first search request text to determine whether to produce a test question meeting the search intent corresponding to the first search request text.

Description

Technical Field

[0001] This disclosure relates to the field of computer technology, and more particularly to the fields of smart education, artificial intelligence, and big data technology, specifically to a data processing method, apparatus, electronic device, computer-readable storage medium, and computer program product. Background Technology

[0002] Education has always been a tradition in China. With the development of science and technology, the popularization of internet information, artificial intelligence, and big data, the demand for online education is increasing. Online education cannot do without a large test question resource library, but there are still situations where the test question resource library cannot cover users' search requests, so test questions need to be produced for these search requests.

[0003] The methods described in this section are not necessarily methods that had been previously conceived or adopted. Unless otherwise specified, no method described in this section should be assumed to be prior art simply because it is included in this section. Similarly, unless otherwise specified, the issues mentioned in this section should not be considered to be accepted in any prior art. Summary of the Invention

[0004] This disclosure provides a data processing method, apparatus, electronic device, computer-readable storage medium, and computer program product.

[0005] According to one aspect of this disclosure, a data processing method is provided, comprising: acquiring a plurality of first text data, wherein the plurality of first text data includes a plurality of first search request texts and a plurality of preset test question texts in a preset test question bank, the plurality of first search request texts including search request texts in historical search logs used to search for test questions and which did not obtain recall results in the preset test question bank, and each of the plurality of first search request texts including identification information; encoding each of the plurality of first text data to determine a plurality of signature values ​​for each of the plurality of first text data; for each of the plurality of first text data, copying the first text data into multiple copies corresponding to the plurality of signature values ​​respectively to obtain a plurality of second text data; and based on the plurality of second text data... This data, along with the signature value corresponding to each second text data, is used to obtain multiple data sets, wherein the signature values ​​of the second text data contained in each of the multiple data sets are the same; for each data set containing the first search request text in the multiple data sets, a first identification result is determined for each first search request text in the data set, the first identification result indicating whether the data set contains a preset test question text with a similarity greater than a first preset threshold to the first search request text; and based on the identification information of each first search request text in the multiple data sets and the first identification result, a second identification result is determined for each first search request text in the multiple first search request texts, so as to determine whether it is necessary to generate test questions that satisfy the search intent corresponding to the first search request text based on the second identification result.

[0006] According to another aspect of this disclosure, a data processing apparatus is provided, comprising: a first acquisition unit configured to acquire a plurality of first text data, wherein the plurality of first text data includes a plurality of first search request texts and a plurality of preset test question texts in a preset test question bank, the plurality of first search request texts including search request texts in historical search logs for searching test questions and for which no recall result was obtained in the preset test question bank, and each of the plurality of first search request texts including identification information; a first determination unit configured to encode each of the plurality of first text data to determine a plurality of signature values ​​for each of the plurality of first text data; a second acquisition unit configured to copy each of the plurality of first text data into multiple copies corresponding to the plurality of signature values ​​to acquire a plurality of second text data; and a third acquisition unit. The system is configured to acquire multiple data sets based on multiple second text data and the signature value corresponding to each second text data, wherein the signature values ​​of the second text data contained in each of the multiple data sets are the same; the second determining unit is configured to determine a first identification result for each first search request text in each of the multiple data sets containing the first search request text, wherein the first identification result indicates whether the data set contains a preset test question text whose similarity to the first search request text is greater than a first preset threshold; and the third determining unit is configured to determine a second identification result for each first search request text in the multiple first search request text based on the identification information of each first search request text in the multiple data sets and the first identification result, so as to determine whether it is necessary to generate test questions that satisfy the search intent corresponding to the first search request text based on the second identification result.

[0007] According to another aspect of this disclosure, an electronic device is provided, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the data processing method described above.

[0008] According to another aspect of this disclosure, a non-transitory computer-readable storage medium is provided storing computer instructions, wherein the computer instructions are used to cause a computer to perform the above-described data processing method.

[0009] According to another aspect of this disclosure, a computer program product is provided, including a computer program, wherein the computer program implements the above-described data processing method when executed by a processor.

[0010] According to one or more embodiments of this disclosure, by calculating multiple signature values ​​of the first search request text and the preset test question text, search requests and test question texts with a high probability of matching can be grouped together by the signature values, and similarity calculation is performed within the group. This avoids a large number of similarity calculations for search requests and test question texts with low similarity probability (each signature value is different), and eliminates the need to compare the similarity between the search request and each test question text, greatly reducing the amount of computation and improving the computational efficiency of determining whether the search request text is a clue to the test question to be produced.

[0011] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this disclosure, nor is it intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0012] The accompanying drawings exemplify embodiments and form part of the specification, serving together with the textual description to explain exemplary implementations of the embodiments. The illustrated embodiments are for illustrative purposes only and do not limit the scope of the claims. Throughout the drawings, the same reference numerals refer to similar but not necessarily identical elements.

[0013] Figure 1 A schematic diagram of an exemplary system in which the various methods described herein may be implemented according to embodiments of the present disclosure is shown;

[0014] Figure 2 A flowchart of a data processing method according to an embodiment of the present disclosure is shown;

[0015] Figure 3 A flowchart illustrating the determination of multiple signature values ​​for each first text data according to embodiments of the present disclosure is shown;

[0016] Figure 4 A flowchart of a data processing method according to an exemplary embodiment of the present disclosure is shown;

[0017] Figure 5 A structural block diagram of a data processing apparatus according to an embodiment of the present disclosure is shown;

[0018] Figure 6 A structural block diagram of an exemplary electronic device that can be used to implement embodiments of the present disclosure is shown. Detailed Implementation

[0019] The exemplary embodiments of this disclosure are described below with reference to the accompanying drawings, including various details of the embodiments to aid understanding, and should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope of this disclosure. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0020] In this disclosure, unless otherwise stated, the use of terms such as "first," "second," etc., to describe various elements is not intended to limit the positional, temporal, or importance relationships of these elements; such terms are merely used to distinguish one element from another. In some examples, the first element and the second element may refer to the same instance of that element, while in other cases, based on the context, they may refer to different instances.

[0021] The terminology used in the description of the various examples described in this disclosure is for the purpose of describing particular examples only and is not intended to be limiting. Unless the context explicitly indicates otherwise, an element may be one or more unless the number of elements is specifically limited. Furthermore, the term "and / or" as used in this disclosure covers any one of the listed items and all possible combinations thereof.

[0022] The embodiments of this disclosure will now be described in detail with reference to the accompanying drawings.

[0023] Figure 1 A schematic diagram of an exemplary system 100 in which the various methods and apparatus described herein can be implemented according to embodiments of this disclosure is shown. Reference Figure 1 The system 100 includes one or more client devices 101, 102, 103, 104, 105 and 106, a server 120, and one or more communication networks 110 coupling the one or more client devices to the server 120. The client devices 101, 102, 103, 104, 105 and 106 can be configured to execute one or more applications.

[0024] In embodiments of this disclosure, server 120 may run one or more services or software applications that enable the execution of the data processing methods of this disclosure.

[0025] In some embodiments, server 120 may also provide other services or software applications, which may include non-virtual and virtual environments. In some embodiments, these services may be provided as web-based services or cloud services, such as to users of client devices 101, 102, 103, 104, 105, and / or 106 under a Software as a Service (SaaS) model.

[0026] exist Figure 1 In the configuration shown, server 120 may include one or more components that implement the functions performed by server 120. These components may include software components, hardware components, or combinations thereof that can be executed by one or more processors. Users operating client devices 101, 102, 103, 104, 105, and / or 106 can sequentially interact with server 120 using one or more client applications to utilize the services provided by these components. It should be understood that various different system configurations are possible and may differ from system 100. Therefore, Figure 1 This is an example of a system used to implement the various methods described herein, and is not intended to be limiting.

[0027] Users can use client devices 101, 102, 103, 104, 105, and / or 106 to input search request text and search for test questions. The client devices can provide an interface that allows users to interact with the client devices. The client devices can also output information to the user through this interface. Although... Figure 1 Only six client devices are described, but those skilled in the art will understand that this disclosure can support any number of client devices.

[0028] Client devices 101, 102, 103, 104, 105, and / or 106 may include various types of computer devices, such as portable handheld devices, general-purpose computers (such as personal computers and laptops), workstation computers, wearable devices, smart screen devices, self-service terminal devices, service robots, gaming systems, thin clients, various messaging devices, sensors, or other sensing devices. These computer devices can run various types and versions of software applications and operating systems, such as Microsoft Windows, Apple iOS, UNIX-like operating systems, Linux or Linux-like operating systems (such as Google Chrome OS); or include various mobile operating systems, such as Microsoft Windows Mobile OS, iOS, Windows Phone, and Android. Portable handheld devices may include cellular phones, smartphones, tablets, personal digital assistants (PDAs), etc. Wearable devices may include head-mounted displays (such as smart glasses) and other devices. Gaming systems may include various handheld gaming devices, internet-enabled gaming devices, etc. Client devices are capable of executing various applications, such as various internet-related applications, communication applications (such as email applications), short message service (SMS) applications, and can use various communication protocols.

[0029] Network 110 can be any type of network well known to those skilled in the art, and can use any of a variety of available protocols (including but not limited to TCP / IP, SNA, IPX, etc.) to support data communication. By way of example only, one or more networks 110 can be a local area network (LAN), an Ethernet-based network, a token ring network, a wide area network (WAN), the Internet, a virtual network, a virtual private network (VPN), an intranet, an extranet, a blockchain network, a public switched telephone network (PSTN), an infrared network, a wireless network (e.g., Bluetooth, WIFI), and / or any combination of these and / or other networks.

[0030] Server 120 may include one or more general-purpose computers, special-purpose server computers (e.g., PC (personal computer) servers, UNIX servers, mid-range servers), blade servers, mainframe computers, server clusters, or any other suitable arrangement and / or combination. Server 120 may include one or more virtual machines running a virtual operating system, or other computing architectures involving virtualization (e.g., one or more flexible pools of logical storage devices that can be virtualized to maintain virtual storage devices for servers). In various embodiments, server 120 may run one or more services or software applications that provide the functionality described below.

[0031] The computing unit in server 120 can run one or more operating systems, including any of the aforementioned operating systems and any commercially available server operating system. Server 120 can also run any of a variety of additional server applications and / or middleware applications, including HTTP servers, FTP servers, CGI servers, JAVA servers, database servers, etc.

[0032] In some implementations, server 120 may include one or more applications to analyze and merge data feeds and / or event updates received from users of client devices 101, 102, 103, 104, 105 and / or 106. Server 120 may also include one or more applications to display data feeds and / or real-time events via one or more display devices of client devices 101, 102, 103, 104, 105 and / or 106.

[0033] In some implementations, server 120 can be a server for a distributed system or a server integrated with blockchain. Server 120 can also be a cloud server, or an intelligent cloud computing server or intelligent cloud host with artificial intelligence technology. A cloud server is a host product in the cloud computing service system, designed to address the shortcomings of traditional physical hosts and Virtual Private Server (VPS) services, such as high management difficulty and weak business scalability.

[0034] System 100 may also include one or more databases 130. In some embodiments, these databases may be used to store data and other information. For example, one or more of the databases 130 may be used to store information such as audio files and video files. Databases 130 may reside in various locations. For example, a database used by server 120 may be local to server 120, or it may be located away from server 120 and may communicate with server 120 via a network-based or dedicated connection. Databases 130 may be of different types. In some embodiments, the database used by server 120 may be, for example, a relational database. One or more of these databases may store, update, and retrieve data from and from the databases in response to commands.

[0035] In some embodiments, one or more of the databases 130 may also be used by an application to store application data. The databases used by the application may be of different types, such as key-value stores, object stores, or regular stores supported by a file system.

[0036] Figure 1 The system 100 can be configured and operated in various ways to enable the application of the various methods and apparatus described in this disclosure.

[0037] According to some embodiments of this disclosure, such as Figure 2 As shown, a data processing method is provided, including:

[0038] Step S201: Obtain multiple first text data, wherein the multiple first text data includes multiple first search request texts and multiple preset test question texts in the preset test question bank, the multiple first search request texts include search request texts in the historical search logs used to search for test questions and which did not obtain recall results in the preset test question bank, and each of the multiple first search request texts includes identification information.

[0039] Step S202: Encode each of the multiple first text data to determine multiple signature values ​​for each first text data.

[0040] Step S203: For each of the multiple first text data, copy the first text data into multiple copies corresponding to multiple signature values ​​to obtain multiple second text data;

[0041] Step S204: Based on multiple second text data and the signature value corresponding to each second text data, obtain multiple data sets, wherein the signature values ​​of the second text data contained in each of the multiple data sets are the same;

[0042] Step S205: For each data set containing the first search request text in multiple data sets, determine a first recognition result for each first search request text in that data set. The first recognition result indicates whether the data set contains a preset test question text with a similarity greater than a first preset threshold to the first search request text; and

[0043] Step S206: Based on the identification information and first identification result of each first search request text in multiple datasets, determine the second identification result of each first search request text in multiple first search request texts, so as to determine whether it is necessary to generate test questions that satisfy the search intent corresponding to the first search request text based on the second identification result.

[0044] Therefore, by calculating multiple signature values ​​of the first search request text and the preset test question text, search requests and test question texts with a high probability of matching can be grouped together by signature values, and similarity calculation can be performed within the group. This avoids a large number of similarity calculations for search requests and test question texts with low similarity probability (each signature value is different), and eliminates the need to compare the similarity between the search request and each test question text, greatly reducing the amount of computation and improving the computational efficiency of determining whether the search request text is a clue to the test question to be produced.

[0045] In some embodiments, the historical search logs of the search engine can be obtained first, and the search request texts used to search for test questions that did not yield recall results in the preset test question bank can be used as the first search request text.

[0046] In some embodiments, the historical search log can be the full historical search log or the historical search log within a preset time range (e.g., within one day).

[0047] In some embodiments, historical search logs can be retrieved at a certain time period (e.g., one day) and the above data processing method can be executed.

[0048] In some embodiments, after obtaining historical search logs, search request texts that did not yield recall results can be identified first. For example, it can be determined whether the search request text has yielded recall results by judging whether the top-ranked recall results for the corresponding search request text contain questions from a preset question bank. Subsequently, the search request texts that did not yield recall results can be analyzed and identified to determine whether the search request text was used to search for questions and / or whether the search request text was used to search for questions within a preset range. The preset range can be a preset educational stage (e.g., primary school, high school, kindergarten to high school graduation, etc.), a preset subject, etc., and the search request texts used to search for questions within the preset range are taken as the first search request texts.

[0049] In some embodiments, before analyzing and identifying the search request text, a search request integrity judgment can be performed on the search request text. For example, the search request text can be identified by regular expressions or by a large language model to determine that the search request text is a semantically complete search request text. This can further filter out some abnormal search request texts, further reduce the amount of subsequent data processing, and improve computational efficiency.

[0050] In some embodiments, the above-described screening and identification process can be set by relevant technical personnel according to actual needs, and is not limited here.

[0051] In some embodiments, obtaining multiple first search request texts may further include: obtaining multiple fourth search request texts from historical search logs that were used to search for test questions and did not yield recall results in the preset test question bank; and calculating the pairwise similarity between the multiple fourth search request texts to obtain multiple first search request texts after deduplication. Thus, the overall computational load can be further reduced through preliminary deduplication.

[0052] In some embodiments, after obtaining the search request text for searching test questions and obtaining no recall results in the preset test question bank, the search request text can be further deduplicated by calculating the pairwise similarity between the search request texts.

[0053] In some embodiments, the similarity between two search request texts can be determined by performing text encoding or semantic encoding on the search request text, converting the search request text into an encoded vector, and calculating the distance or similarity between the encoded vectors. When the similarity is greater than a certain preset threshold, the two search request texts can be considered to have the same search intent, and they can be deduplicated to obtain multiple first search request texts.

[0054] In some embodiments, each first search request text includes its corresponding identification information.

[0055] In some embodiments, the identification information of each first search request text can be obtained based on a message digest algorithm or text encoding. For example, the MD4 message digest algorithm or the MD5 message digest algorithm can be applied to process the first search request text to obtain the identification information corresponding to the first search request text.

[0056] In some embodiments, the preset question bank may be a pre-built question bank for a preset educational stage and / or a preset subject, which stores a massive amount of question resources. For example, a question bank for the K-12 education stage (from kindergarten to high school graduation) may contain 600 million questions.

[0057] In some embodiments, the preset question text may be the stem of each question, or the stem of each question and all options.

[0058] In some embodiments, test questions in the test question bank may be stored as images. For these test questions, text recognition can be performed on the images first, and the recognized text can be organized into preset test question text according to a preset format.

[0059] In some embodiments, the first search request text and the preset test question text may be stored in different files or in the same file. When the first search request text and the preset test question text are stored in the same file, they can be distinguished by a resource type identifier.

[0060] After obtaining text data such as the first search request text and the preset test question text, each piece of text data can be encoded to map each piece of text data into an encoding vector, and multiple signature values ​​can be determined based on the encoding vector.

[0061] In some embodiments, text data can be mapped to encoded vectors using a language model, a text encoding model, or preset encoding rules.

[0062] In some embodiments, the encoding vector can be segmented so that each segment of the encoding vector is used as a signature value; in some embodiments, the text data can also be segmented first and encoded separately so that the resulting multiple encoding vectors are used as multiple signature values.

[0063] In some embodiments, such as Figure 3 As shown, encoding processing is performed on each of the multiple first text data to determine the multiple signature values ​​of each first text data, which may include:

[0064] For each piece of text data in a set of multiple text data, perform the following operation:

[0065] Step S301: Preprocess the text data to obtain multiple word segments of the text data. The preprocessing includes at least punctuation replacement and word segmentation.

[0066] Step S302: Encode multiple word segments to obtain multiple word segment codes corresponding to multiple word segments;

[0067] Step S303: Based on multiple word segmentation codes, determine the first signature value corresponding to the text data; and

[0068] Step S304: Segment the first signature value to determine multiple signature values ​​of the text data.

[0069] Therefore, by performing preprocessing such as word segmentation on each piece of text data and encoding each word separately, the accuracy of signature value calculation can be improved, thereby improving the accuracy of subsequent grouping.

[0070] In some embodiments, text data may be preprocessed before encoding. Preprocessing may include punctuation replacement and word segmentation.

[0071] In some embodiments, preprocessing may further include stop word replacement and other processing. It is understood that those skilled in the art can set other preprocessing methods as needed, and no restrictions are imposed here.

[0072] In some embodiments, word segmentation can be performed using methods such as Jieba segmentation and NGRAM segmentation (e.g., 3Gram segmentation, 4Gram segmentation).

[0073] For example, for the preset test text "In the two-person competition of the Little Singer Competition, Li Hui and Wang Le scored a total of 8.46 points. It is known that Li Hui scored 0.46 points more than Wang Le. How many points did they each score?", we can first replace the punctuation marks to obtain the text "In the two-person competition of the Little Singer Competition, Li Hui and Wang Le scored a total of 846 points. It is known that Li Hui scored 0.46 points more than Wang Le. How many points did they each score?"; then, through 3Gram segmentation, we obtain the following multiple segmented words: ['in the Little Singer', 'Little Singer', 'Singer Competition', 'Competition', 'Two', 'Two People', 'Two People', 'People Group', 'Group Competition', 'Competition', 'Competition' In the middle, the competition is Li, Li Hui, Li Hui and, Hui and Wang, and Wang Le, Wang Le one, Le group, group total, group total score, total score, score 8, 84, 846, 46 points, 6 points already, points known, known Li, known Li Hui, Li Hui than, Hui than Wang, than Wang Le, Wang Le more, Le more score, more score 0, score 04, 046, 46 points, 6 points then, points then two, then two people, two people score, people respectively, respectively score, other scores, score more, how many points.

[0074] In some embodiments, each word segment can be encoded separately, and the encodings of multiple word segments can be weighted or concatenated to obtain a first signature value.

[0075] In some embodiments, a hash function can be applied to each word to obtain its segmentation encoding. In some embodiments, a text encoding model or a language model can also be applied to encode each word.

[0076] In some embodiments, after applying a hash function to map each word, a multi-bit binary string (i.e., word segmentation code) corresponding to each word can be obtained; then, multiple word segmentation codes can be weighted and summed to obtain a merged binary string, which can be used as the first signature value.

[0077] In some embodiments, the merged binary string may be further reduced in dimensionality to map it to a 64-bit binary string, which is then used as the first signature value.

[0078] In some exemplary embodiments, the SimHash algorithm can be applied to calculate a first signature value for each text data.

[0079] In some embodiments, multiple signature values ​​can be obtained by splitting the first signature value. For example, a first signature value can be divided into four signature values.

[0080] For example, if a certain text data is processed by the SimHash algorithm and its first signature value is [32714190792408211537], then the four signature values ​​of the text data can be [32714, 19079, 24082, 11537].

[0081] Subsequently, for each of the multiple first text data sets, based on its multiple signature values, the first text data set can be copied multiple times, with each copy corresponding to one of the signature values. By performing the above copying process on each first text data set, multiple second text data sets are obtained.

[0082] Furthermore, multiple data sets can be obtained based on multiple second text data and the signature value corresponding to each second text data, wherein the signature value of the second text data contained in each of the multiple data sets is the same.

[0083] In some embodiments, each of the plurality of signature values ​​corresponds to a sequence number, the sequence number indicating the position of the signature value in the first signature value corresponding to that signature value. Obtaining a plurality of data sets based on a plurality of second text data and the signature value corresponding to each second text data includes: obtaining a plurality of data sets based on a plurality of second text data and the signature value and sequence number corresponding to each second text data, wherein the signature value and sequence number of the second text data contained in each of the plurality of data sets are the same.

[0084] Therefore, by using sequential numbering, the same signature value in different positions in the signature value can be further distinguished, thereby refining the grouping, improving the coverage of the grouping, and further improving the computational efficiency.

[0085] For example, if a certain text data is processed using the SimHash algorithm and its first signature value is [32714190792408211537], then the four signature values ​​of the text data can be [32714, 19079, 24082, 11537], and each signature value has a sequential order. The sequential number of signature value '32714' can be '0', the sequential number of signature value '19079' can be '1', the sequential number of signature value '24082' can be '2', and the sequential number of signature value '11537' can be '3'.

[0086] When constructing a dataset, both the signature value and the sequence number can be used as the criteria for grouping text data, thereby further improving the coverage of the dataset and ensuring that the search request text can be similar to more questions with a high probability of matching while reducing the amount of data processing.

[0087] Subsequently, for each of the multiple datasets containing the first search request text, a first identification result is determined for each first search request text in that dataset. The first identification result can be used to indicate whether the dataset contains preset test question texts with a similarity greater than a first preset threshold to the first search request text (i.e., whether the dataset contains test question texts that can satisfy the search intent of the first search request text).

[0088] In some embodiments, the similarity between two search request texts can be determined by performing text encoding or semantic encoding on the search request text and the preset test question text, converting the text data into encoded vectors, and calculating the distance or similarity between the encoded vectors.

[0089] In some embodiments, when a dataset contains only preset test question text, the text in that dataset may not be processed.

[0090] In some embodiments, when a dataset contains only search request text, each search request text in the dataset cannot be matched with a preset test question text whose similarity is greater than a first preset threshold. Therefore, the first identification result of each search request text in the dataset can be directly determined as the dataset not containing a preset test question text whose similarity to the first search request text is greater than the first preset threshold.

[0091] After determining the first identification result of the search request text in all datasets, these identification results can be integrated to determine the second identification result of each of the multiple first search request texts. The second identification result can then be used to indicate whether the first search request text requires test item generation.

[0092] In some embodiments, multiple first identification results corresponding to each search request text in different datasets can be aggregated according to the identification information corresponding to the search request text, thereby obtaining a second identification result for the search request text. For example, if the search request text cannot be matched with a preset test question text with a similarity greater than a first preset threshold in each dataset, it is considered that the search request text does not have a matching test question in the preset test question library, and it is identified as a test question clue to be generated; otherwise, it is considered that it can be matched with a test question that satisfies its search intent in the preset test question library, and it will no longer be used as a test question clue for test question generation.

[0093] In some embodiments, the above method can be further combined with a distributed computing framework (such as Hadoop's MapReduce computing framework) to execute.

[0094] In some exemplary embodiments, before preprocessing the text data, all the first text data can be organized into a plain text file according to a preset format, and the file can be uploaded to a distributed file system for storage.

[0095] In some embodiments, the preset format may be, for example, that each first text data corresponds to one line in the file, and each line includes the identification information, resource type information, and the text data itself corresponding to the first text data, and is connected by delimiters. It is understood that those skilled in the art can also determine the preset format according to actual needs, and no limitation is imposed here.

[0096] In some embodiments, during the Map phase of the MapReduce computing framework, each first text data in the file system can be preprocessed, its signature value calculated, and the text data copied, and then grouped into multiple data sets according to the signature value (or the signature value and the sequential numbering of the signature values).

[0097] In some embodiments, text data from the same dataset can be distributed to a computing node in a distributed computing system during the Reduce phase of the MapReduce computing framework to achieve distributed computing and further improve computing efficiency.

[0098] In some embodiments, the data processing method may further include: in response to a second identification result of a second search request text among a plurality of first search request texts indicating the need to generate test questions that satisfy the search intent corresponding to the first search request text, determining the similarity between the second search request text and each existing test question clue text in the test question clue database; and in response to the similarity between the second search request text and each existing test question clue text in the test question clue database being less than a second preset threshold, storing the second search request text in the test question clue database.

[0099] Therefore, after determining that the search request text is a clue for test questions to be produced, it can be compared with existing test question clues to remove duplicates before entering the database, thereby further avoiding the subsequent duplication of test question production.

[0100] In some embodiments, storing the second search request text into the test question clue database includes: identifying the second search request text to determine the classification information of the second search request text, the classification information including at least one of test subject and test grade level; and storing the second search request text and the classification information into the test question clue database.

[0101] Therefore, before the test questions are entered into the database, the test question clues are classified by subject and / or grade level, and the classification information is recorded together, thereby further improving the efficiency of subsequent test question production.

[0102] In some embodiments, the above data processing method may further include: in response to a second identification result indication from a third search request text among the plurality of first search request texts indicating that it is not necessary to generate test questions that satisfy the search intent corresponding to the first search request text, adding the third search request text to the index information corresponding to the target test question text among the plurality of preset test question texts, so as to re-push the updated index information to the search service, wherein the similarity between the target test question text and the third search request text is less than or equal to a third preset threshold.

[0103] Therefore, for cases where there are already corresponding questions in the question bank, but the search engine's retrieval results are poor, the index information of the question can be optimized by adding the search request text to the index information of the corresponding question and re-pushing it to the search service, thereby improving the subsequent retrieval of the question.

[0104] In some embodiments, during the similarity comparison process for grouping, if a search request text matches a preset test question text, the index information of the test question corresponding to that preset test question text is considered defective, thus preventing it from being retrieved during the search. Therefore, for the aforementioned test question text, it can be recorded, its index information optimized (e.g., its corresponding search request text added to its index information), and then resubmitted to the search service to optimize subsequent search results.

[0105] In some embodiments, each preset test question text may also correspond to an identification information. When summarizing the recognition results, the above information may also be summarized according to the identification information of each test question.

[0106] Figure 4 A flowchart of a data processing method according to an exemplary embodiment of the present disclosure is shown.

[0107] In some exemplary embodiments, such as Figure 4 As shown, data processing methods may include:

[0108] Step S401: Obtain the search request text that did not yield recall results from the historical search log;

[0109] Step S402: Analyze and identify the search request texts that have not yielded recall results, in order to filter the search request texts used to search for test questions and for searching for test questions in the preset learning stage;

[0110] Step S403: Apply the SimHash algorithm to deduplicate the filtered search request texts to obtain multiple first search request texts.

[0111] Step S404: Push multiple first search request texts to the distributed file system;

[0112] Step S405: Based on the Hadoop MapReduce computing framework, perform preprocessing, grouping and similarity matching operations on text data, and summarize the recognition results;

[0113] Step S406: Determine whether each first search request text has a preset test question text with a similarity greater than a first preset threshold (e.g., 0.9);

[0114] Step S407: When available, optimize the index information of the corresponding preset test questions and push them back to the search service;

[0115] Step S408: If not available, output the search request text to the production thread file;

[0116] Step S409: Classify each search request text in the production lead file by subject and grade level;

[0117] Step S410: Calculate the similarity between each search request text and the existing test clues in the test clue database, and filter out search request texts with a similarity less than the second preset threshold (e.g., 0.9).

[0118] Step S411: Store the search request texts and their classification information that meet the above conditions into the test clue database; and

[0119] Step S412: Receive unproduced test question clues from the test question clue database to produce test questions.

[0120] Therefore, by using information from search logs, we can efficiently and accurately identify clues for producing test questions while avoiding the duplication of test question production.

[0121] In some embodiments, such as Figure 5 As shown, a data processing apparatus 500 is provided, comprising:

[0122] The first acquisition unit 510 is configured to acquire multiple first text data, wherein the multiple first text data includes multiple first search request texts and multiple preset test question texts in the preset test question library, the multiple first search request texts include search request texts in the historical search logs used to search for test questions and which did not obtain recall results in the preset test question library, and each of the multiple first search request texts includes identification information.

[0123] The first determining unit 520 is configured to encode each of the plurality of first text data to determine the plurality of signature values ​​of each first text data.

[0124] The second acquisition unit 530 is configured to copy each of the multiple first text data into multiple copies corresponding to multiple signature values, in order to acquire multiple second text data.

[0125] The third acquisition unit 540 is configured to acquire multiple data sets based on multiple second text data and the signature value corresponding to each second text data, wherein the signature value of the second text data contained in each of the multiple data sets is the same;

[0126] The second determining unit 550 is configured to, for each of a plurality of data sets containing the first search request text, determine a first identification result for each first search request text in that data set, wherein the first identification result indicates whether the data set contains a preset test question text whose similarity to the first search request text is greater than a first preset threshold; and

[0127] The third determining unit 560 is configured to determine a second identification result for each first search request text in the plurality of first search request texts based on the identification information and the first identification result of each first search request text in the plurality of data sets, so as to determine whether it is necessary to generate test questions that satisfy the search intent corresponding to the first search request text based on the second identification result.

[0128] Therefore, by calculating multiple signature values ​​of the first search request text and the preset test question text, search requests and test question texts with a high probability of matching can be grouped together by signature values, and similarity calculation can be performed within the group. This avoids a large number of similarity calculations for search requests and test question texts with low similarity probability (each signature value is different), and eliminates the need to compare the similarity between the search request and each test question text, greatly reducing the amount of computation and improving the computational efficiency of determining whether the search request text is a clue to the test question to be produced.

[0129] In some embodiments, the first determining unit may be configured to perform the operations of the following sub-units for each of a plurality of text data. The first determining unit may include: a preprocessing sub-unit configured to preprocess the text data to obtain a plurality of word segments of the text data, the preprocessing including at least punctuation replacement and word segmentation processing; an encoding sub-unit configured to encode the plurality of word segments to obtain a plurality of word segmentation codes corresponding to the plurality of word segments; a first determining sub-unit configured to determine a first signature value corresponding to the text data based on the plurality of word segmentation codes; and a second determining sub-unit configured to segment the first signature value to determine a plurality of signature values ​​of the text data.

[0130] In some embodiments, each of the plurality of signature values ​​corresponds to a sequence number, the sequence number indicating the position of the signature value in the first signature value corresponding to that signature value. The third acquisition unit may be further configured to: acquire a plurality of data sets based on a plurality of second text data and the signature value and sequence number corresponding to each of the second text data, wherein the signature value and sequence number of the second text data contained in each of the plurality of data sets are the same.

[0131] In some embodiments, the data processing apparatus may further include: a fourth determining unit configured to determine the similarity between the second search request text and each existing test question clue text in the test question clue database in response to a second identification result of a second search request text among a plurality of first search request texts indicating that test questions that satisfy the search intent corresponding to the first search request text need to be generated; and a storage unit configured to store the second search request text in the test question clue database in response to a similarity between the second search request text and each existing test question clue text in the test question clue database being less than a second preset threshold.

[0132] In some embodiments, the storage unit may include: a third determining subunit configured to identify the second search request text to determine classification information of the second search request text, the classification information including at least one of the subject and grade level of the test questions; and a storage subunit configured to store the second search request text and the classification information into a test question clue database.

[0133] In some embodiments, the data processing apparatus may further include: an adding unit configured to, in response to a second identification result indication from a third search request text among a plurality of first search request texts, not requiring the generation of test questions that satisfy the search intent corresponding to the first search request text, add the third search request text to index information corresponding to a target test question text among a plurality of preset test question texts, so as to re-push the updated index information to the search service, wherein the similarity between the target test question text and the third search request text is less than or equal to a third preset threshold.

[0134] In some embodiments, the first acquisition unit may include: a first acquisition subunit configured to acquire multiple fourth search request texts from historical search logs; and a deduplication subunit configured to calculate the pairwise similarity between the multiple fourth search request texts, so as to obtain multiple first search request texts after deduplicating the multiple fourth search request texts.

[0135] According to embodiments of this disclosure, an electronic device, a readable storage medium, and a computer program product are also provided.

[0136] refer to Figure 6The present invention describes a structural block diagram of an electronic device 600 that can serve as a server or client of the present disclosure, which is an example of a hardware device that can be applied to various aspects of the present disclosure. The electronic device is intended to represent various forms of digital electronic computer devices, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0137] like Figure 6 As shown, the electronic device 600 includes a computing unit 601, which can perform various appropriate actions and processes based on a computer program stored in a read-only memory (ROM) 602 or a computer program loaded from a storage unit 608 into a random access memory (RAM) 603. The RAM 603 may also store various programs and data required for the operation of the electronic device 600. The computing unit 601, ROM 602, and RAM 603 are interconnected via a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.

[0138] Multiple components in electronic device 600 are connected to I / O interface 605, including: input unit 606, output unit 607, storage unit 608, and communication unit 609. Input unit 606 can be any type of device capable of inputting information to electronic device 600. Input unit 606 can receive input digital or character information and generate key signal inputs related to user settings and / or function control of electronic device, and can include, but is not limited to, a mouse, keyboard, touchscreen, trackpad, trackball, joystick, microphone, and / or remote control. Output unit 607 can be any type of device capable of presenting information, and can include, but is not limited to, a monitor, speaker, video / audio output terminal, vibrator, and / or printer. Storage unit 608 can include, but is not limited to, disk and optical disk. Communication unit 609 allows electronic device 600 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks, and can include, but is not limited to, modems, network cards, infrared communication devices, wireless communication transceivers, and / or chipsets, such as Bluetooth devices, 802.11 devices, WiFi devices, WiMax devices, cellular communication devices, and / or the like.

[0139] The computing unit 601 can be various general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 601 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 601 performs the various methods and processes described above, such as the data processing methods given in some embodiments of this disclosure. For example, in some embodiments, the data processing methods given in some embodiments of this disclosure can be implemented as computer software programs tangibly contained in a machine-readable medium, such as storage unit 608. In some embodiments, part or all of the computer program can be loaded and / or installed on the electronic device 600 via ROM 602 and / or communication unit 609. When the computer program is loaded into RAM 603 and executed by the computing unit 601, one or more steps of the data processing methods given in some embodiments of this disclosure can be performed. Alternatively, in other embodiments, the computing unit 601 may be configured by any other suitable means (e.g., by means of firmware) to perform the data processing methods given in some embodiments of this disclosure.

[0140] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0141] The program code used to implement the methods of this disclosure may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0142] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0143] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0144] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as a data server), or computing systems that include middleware components (e.g., an application server), or computing systems that include frontend components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with embodiments of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., a communication network). Examples of communication networks include local area networks (LANs), wide area networks (WANs), and the Internet.

[0145] Computer systems can include clients and servers. Clients and servers are generally located far apart and typically interact via communication networks. Client-server relationships are created by computer programs running on the respective computers and having a client-server relationship with each other. Servers can be cloud servers, servers in distributed systems, or servers incorporating blockchain technology.

[0146] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this disclosure can be performed in parallel, sequentially, or in a different order, as long as the desired result of the technical solution disclosed in this disclosure can be achieved, and this is not limited herein.

[0147] While embodiments or examples of this disclosure have been described with reference to the accompanying drawings, it should be understood that the methods, systems, and devices described above are merely exemplary embodiments or examples, and the scope of the invention is not limited by these embodiments or examples, but only by the granted claims and their equivalents. Various elements in the embodiments or examples may be omitted or replaced by their equivalents. Furthermore, the steps may be performed in a different order than that described in this disclosure. Further, various elements in the embodiments or examples may be combined in various ways. Importantly, as the technology evolves, many elements described herein can be replaced by equivalents that appear after this disclosure.

Claims

1. A data processing method, comprising: Acquire multiple first text data, wherein the multiple first text data includes multiple first search request texts and multiple preset test question texts in a preset test question bank, the multiple first search request texts include search request texts in historical search logs used to search for test questions and for which no recall results were obtained in the preset test question bank, and each of the multiple first search request texts includes identification information; Each of the plurality of first text data is encoded to determine multiple signature values ​​for each first text data. The multiple signature values ​​for each first text data are obtained by segmenting the encoding vector corresponding to the first text data, or by segmenting the first text data and encoding each segment separately. For each of the plurality of first text data, the first text data is copied into multiple copies corresponding to the plurality of signature values ​​respectively, to obtain a plurality of second text data; Based on the plurality of second text data and the signature value corresponding to each second text data, a plurality of data sets are obtained, wherein the signature values ​​of the second text data contained in each of the plurality of data sets are the same; For each of the plurality of data sets containing the first search request text, a first recognition result is determined for each first search request text in that data set. The first recognition result indicates whether the data set contains a preset test question text whose similarity to the first search request text is greater than a first preset threshold. Based on the identification information and first identification result of each first search request text in the plurality of data sets, a second identification result of each first search request text in the plurality of first search request texts is determined, so as to determine whether it is necessary to generate test questions that satisfy the search intent corresponding to the first search request text based on the second identification result.

2. The method according to claim 1, wherein, The step of encoding each of the plurality of first text data to determine the plurality of signature values ​​for each first text data includes: For each of the plurality of first text data, perform the following operation: The first text data is preprocessed to obtain multiple word segments of the first text data. The preprocessing includes at least punctuation replacement and word segmentation. Encode the multiple word segments to obtain multiple word segmentation codes corresponding to the multiple word segments; Based on the multiple word segmentation codes, determine the first signature value corresponding to the first text data; and The first signature value is segmented to determine multiple signature values ​​of the first text data.

3. The method according to claim 2, wherein, Each of the plurality of signature values ​​corresponds to a sequential number, the sequential number indicating the position of the signature value within the first signature value corresponding to that signature value. The step of obtaining multiple data sets based on the plurality of second text data and the signature value corresponding to each second text data includes: Based on the plurality of second text data and the signature value and sequence number corresponding to each second text data, the plurality of data sets are obtained, wherein the signature value and sequence number of the second text data contained in each of the plurality of data sets are the same.

4. The method according to claim 1, further comprising: In response to a second identification result indicating that a test question needs to be generated that satisfies the search intent corresponding to the first search request text among the plurality of first search request texts, the similarity between the second search request text and each existing test question clue text in the test question clue database is determined; as well as In response to the fact that the similarity between the second search request text and each existing question clue text in the question clue database is less than a second preset threshold, the second search request text is stored in the question clue database.

5. The method according to claim 4, wherein, The step of storing the second search request text into the test question clue database includes: The second search request text is identified to determine its classification information, which includes at least one of the subject matter and the grade level of the test questions; and The second search request text and the classification information are stored in the test question clue database.

6. The method according to claim 1, further comprising: In response to the second recognition result of the third search request text among the plurality of first search request texts indicating that it is not necessary to generate test questions that satisfy the search intent corresponding to the first search request text, the third search request text is added to the index information corresponding to the target test question text among the plurality of preset test question texts, so as to push the updated index information back to the search service, wherein the similarity between the target test question text and the third search request text is less than or equal to a third preset threshold.

7. The method according to any one of claims 1 to 6, wherein, The acquisition of the plurality of first search request texts includes: In the historical search log, obtain multiple fourth search request texts used to search for test questions that did not yield recall results in the preset test question bank; and Calculate the similarity between each pair of the plurality of fourth search request texts, so as to obtain the plurality of first search request texts after deduplication of the plurality of fourth search request texts.

8. A data processing apparatus, comprising: The first acquisition unit is configured to acquire multiple first text data, wherein the multiple first text data includes multiple first search request texts and multiple preset test question texts in a preset test question bank, the multiple first search request texts include search request texts in historical search logs used to search for test questions and which did not obtain recall results in the preset test question bank, and each of the multiple first search request texts includes identification information; The first determining unit is configured to encode each of the plurality of first text data to determine multiple signature values ​​for each first text data. The multiple signature values ​​for each first text data are obtained by segmenting the encoding vector corresponding to the first text data, or by segmenting the first text data and encoding each segment of text data separately. The second acquisition unit is configured to copy each of the plurality of first text data into multiple copies corresponding to the plurality of signature values, thereby acquiring a plurality of second text data. The third acquisition unit is configured to acquire multiple data sets based on the plurality of second text data and the signature value corresponding to each second text data, wherein the signature values ​​of the second text data contained in each of the plurality of data sets are the same; The second determining unit is configured to, for each of the plurality of data sets containing the first search request text, determine a first identification result for each first search request text in that data set, wherein the first identification result indicates whether the data set contains a preset test question text whose similarity to the first search request text is greater than a first preset threshold; and The third determining unit is configured to determine a second identification result for each first search request text in the plurality of data sets based on the identification information and the first identification result of each first search request text in the plurality of data sets, so as to determine whether it is necessary to generate test questions that satisfy the search intent corresponding to the first search request text based on the second identification result.

9. The apparatus according to claim 8, wherein, The first determining unit is configured to perform the following sub-unit operations for each of the plurality of first text data, wherein the first determining unit includes: A preprocessing subunit is configured to preprocess the first text data to obtain multiple word segments of the first text data, wherein the preprocessing includes at least punctuation replacement and word segmentation processing. The encoding subunit is configured to encode the plurality of word segments to obtain the plurality of word segment codes corresponding to the plurality of word segments; The first determining subunit is configured to determine the first signature value corresponding to the first text data based on the plurality of word segmentation codes; and The second determining subunit is configured to segment the first signature value to determine multiple signature values ​​of the first text data.

10. The apparatus according to claim 9, wherein, Each of the plurality of signature values ​​corresponds to a sequential number, the sequential number indicating the position of the signature value within the first signature value corresponding to that signature value; the third acquisition unit is further configured to: Based on the plurality of second text data and the signature value and sequence number corresponding to each second text data, the plurality of data sets are obtained, wherein the signature value and sequence number of the second text data contained in each of the plurality of data sets are the same.

11. The apparatus of claim 8, further comprising: The fourth determining unit is configured to, in response to a second identification result indication of a second search request text among the plurality of first search request texts indicating the need to generate test questions that satisfy the search intent corresponding to the first search request text, determine the similarity between the second search request text and each existing test question clue text in the test question clue database; as well as The storage unit is configured to store the second search request text into the question clue database in response to a second preset threshold being found that the similarity between the second search request text and each existing question clue text in the question clue database is less than a second preset threshold.

12. The apparatus according to claim 11, wherein, The storage unit includes: The third determining subunit is configured to identify the second search request text to determine the classification information of the second search request text, the classification information including at least one of the subject and grade level of the test questions; and The storage subunit is configured to store the second search request text and the classification information into the test question clue database.

13. The apparatus according to claim 8, further comprising: The adding unit is configured to, in response to a second identification result indication from a third search request text among the plurality of first search request texts indicating that it is not necessary to generate a test question that satisfies the search intent corresponding to the first search request text, add the third search request text to the index information corresponding to the target test question text among the plurality of preset test question texts, so as to re-push the updated index information to the search service, wherein the similarity between the target test question text and the third search request text is less than or equal to a third preset threshold.

14. The apparatus according to any one of claims 8 to 13, wherein, The first acquisition unit includes: The first acquisition subunit is configured to acquire, from the historical search log, multiple fourth search request texts for searching for test questions that have not yielded recall results in the preset test question bank; and The deduplication subunit is configured to calculate the similarity between each pair of the plurality of fourth search request texts, so as to obtain the plurality of first search request texts after deduplication of the plurality of fourth search request texts.

15. An electronic device comprising: At least one processor; as well as A memory that is communicatively connected to the at least one processor; in The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-7.

16. A non-transitory computer-readable storage medium storing computer instructions, wherein, The computer instructions are used to cause the computer to perform the method according to any one of claims 1-7.

17. A computer program product comprising a computer program, wherein, The computer program, when executed by a processor, implements the method of any one of claims 1-7.

Images