A text detection method and device
By dividing text into multiple segments and using multiple threads to perform parallel detection, the problem of low text detection efficiency in traditional technology is solved, and the text detection efficiency is improved.
Patent Information
- Application Number
- CN202410325753.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-03-21
- Publication Date
- 2024-11-26
- Estimated Expiration
- 2044-03-21
AI Technical Summary
When traditional technology uses DFA to detect text, serial matching of table rows is achieved through state transfer, resulting in low text detection efficiency.
The text to be detected is divided into multiple text segments, and different text segments are detected in parallel by using multiple threads. By obtaining multiple target characters at one time, untrusted characters and transfer status are determined in parallel, so as to realize parallel detection of text segments and character levels.
It improves text detection efficiency, avoids the slow speed problem caused by serial detection, and solves the problem of low text detection efficiency in traditional technologies.
Smart Images

Figure CN118132676B_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of network security technology, and in particular to a text detection method and device. Background Art
[0002] The Internet has become an indispensable part of people's lives. However, with the continuous development of Internet technology, the frequency of network attacks is also increasing. These network attacks will not only bring a lot of trouble to users, but also endanger the security and stability of the network to a certain extent. In order to prevent the occurrence of network security incidents, network security detection plays a vital role in ensuring network security. Traditional technology usually compiles regular expressions into deterministic finite automatons (DFA) to perform network security detection tasks, that is, the regular expression detection and matching engine converts NFA into DFA, and then DFA matches the network data (i.e., the text to be detected) to achieve network security detection.
[0003] However, when using DFA to detect text, the traditional technology performs serial matching of state transition table rows, that is, performs text detection one character at a time, which leads to the problem of low text detection efficiency. Summary of the invention
[0004] In view of this, the purpose of the present application is to provide a text detection method and device to solve the problem of low text detection efficiency in traditional technologies.
[0005] In a first aspect, an embodiment of the present application provides a text detection method, comprising:
[0006] Divide the text to be detected into multiple text segments, so as to use multiple threads to perform text detection on different text segments at the same time;
[0007] For each text segment, when the target thread is used to detect the text segment, multiple target characters in the text segment are obtained at one time, and it is determined in parallel whether there are untrusted characters among the multiple target characters, where the untrusted characters are characters that can cause the zero state to transfer;
[0008] If there is an untrusted character, for each target character after the first untrusted character among the multiple target characters, determine the transfer state corresponding to the target character;
[0009] If the transfer state is the target state, the process returns to the step of acquiring multiple target characters in the text segment at one time, so as to perform text detection on the newly acquired multiple target characters.
[0010] Optionally, after determining the transfer state corresponding to the target character, it also includes: if the transfer state is not the target state, determining whether the current character is outside a preset range, and the current character is the next character of the character; if the current character is outside the preset range, determining whether there is an acceptable state in the transfer state corresponding to the current character.
[0011] Optionally, multiple target characters in the text segment are obtained at one time, and whether there are untrusted characters among the multiple target characters is determined in parallel, including: obtaining multiple target characters at one time by loading a character block; using a vector comparison instruction to compare the multiple target characters with an untrusted character set in parallel to determine whether there are target characters among the multiple target characters that match the untrusted characters in the untrusted character set; if there are matching target characters, determining that there are untrusted characters among the multiple target characters.
[0012] Optionally, the method also includes: determining whether the ending state of the first text segment is equal to the preset starting state of the second text segment, the first text segment and the second text segment are two adjacent text segments among multiple text segments; if the ending state is equal to the starting state, determining whether the text to be detected hits the rule based on multiple transition states corresponding to the first text segment and the second text segment.
[0013] Optionally, after determining whether the ending state of the first text segment is equal to the preset starting state of the second text segment, it also includes: if the ending state is not equal to the starting state, the second text segment is detected using the first detection path and the second detection path respectively to obtain multiple first path transfer states and multiple second path transfer states, the first detection path is a path that uses the ending state of the first text segment as the starting state of the second text segment to detect characters within a preset range in the second text segment, and the second detection path is a path that detects characters within a preset range in the second text segment when the starting state of the second text segment remains unchanged; determining whether there is an intersection between the multiple first path transfer states and the multiple second path transfer states; if there is an intersection and there is an acceptable state during the detection process, determining a hit rule for the text to be detected; if there is no intersection and there is no acceptable state during the detection process for the first text segment, the ending state of the first text segment is used as the starting state of the second text segment, and text detection is performed on characters outside the preset range in the second text segment according to the starting state.
[0014] Optionally, determining whether multiple first path transfer states intersect with multiple second path transfer states includes: for each first path transfer state, determining whether the second path transfer state at the corresponding position is the same as the first path transfer state; if they are the same, determining that multiple first path transfer states intersect with multiple first path transfer states.
[0015] Optionally, after determining in parallel whether there are untrusted characters in the multiple target characters, the method further includes: if there are no untrusted characters, skipping text detection on the multiple target characters, and selecting multiple characters after the multiple target characters as new multiple target characters to determine whether there are untrusted characters in the new multiple target characters.
[0016] Optionally, whether the text to be detected hits the rule is determined based on multiple transition states corresponding to the first text segment and the second text segment, including: if an acceptable state exists among the multiple transition states corresponding to the first text segment and the second text segment, then it is determined that the text to be detected hits the rule; if no acceptable state exists among the multiple transition states corresponding to the first text segment and the second text segment, then the second text segment is used as a new first text segment, and the text segment after the second text segment is used as a new second text segment, so as to determine whether the text to be detected hits the rule based on the new first text segment and the new second text segment.
[0017] Optionally, after there is an intersection and an acceptable state exists during the detection process, it also includes: if there is an intersection and there is no acceptable state during the detection process, the second text segment is used as a new first text segment, and the text segment after the second text segment is used as a new second text segment, so as to determine whether the text to be detected hits the rule based on the new first text segment and the new second text segment.
[0018] In a second aspect, an embodiment of the present application further provides a text detection device, the device comprising:
[0019] A text segmentation module is used to divide the text to be detected into multiple text segments so as to use multiple threads to perform text detection on different text segments at the same time;
[0020] The untrusted character detection module is used for obtaining multiple target characters in each text segment at one time when detecting the text segment using the target thread, and determining in parallel whether there are untrusted characters among the multiple target characters, wherein the untrusted characters are characters that can cause the zero state to transfer;
[0021] A transfer state determination module, for determining, if there is an untrusted character, for each target character after the first untrusted character among the multiple target characters, a transfer state corresponding to the target character;
[0022] The loop iteration module is used to return to the step of acquiring multiple target characters in the text segment at one time if the transfer state is the target state, so as to perform text detection on the newly acquired multiple target characters.
[0023] The embodiments of the present application bring the following beneficial effects:
[0024] A text detection method and device provided in an embodiment of the present application can use multi-threading to simultaneously detect multiple text segments of the text to be detected, avoiding the problem of slow text detection caused by serial detection of each character in turn. At the same time, it can also perform parallel detection on multiple target characters read at one time, so as to accelerate parallel detection at both the text segment and character levels, further improving the text detection efficiency. Compared with the text detection method in the traditional technology, it solves the problem of low text detection efficiency in the traditional technology.
[0025] In order to make the above-mentioned objects, features and advantages of the present application more obvious and easy to understand, preferred embodiments are specifically cited below and described in detail with reference to the attached drawings. BRIEF DESCRIPTION OF THE DRAWINGS
[0026] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments will be briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.
[0027] Figure 1 A flowchart of a text detection method provided in an embodiment of the present application is shown;
[0028] Figure 2 A schematic diagram of a state transition table provided in an embodiment of the present application is shown;
[0029] Figure 3 A flow chart showing a method for determining that untrusted characters exist among a plurality of target characters provided by an embodiment of the present application is shown;
[0030] Figure 4 A flowchart of a text detection method when the transfer state is not the target state provided by an embodiment of the present application is shown;
[0031] Figure 5 A flow chart of a state detection method provided in an embodiment of the present application is shown;
[0032] Figure 6 A schematic diagram of the structure of a text detection device provided in an embodiment of the present application is shown. DETAILED DESCRIPTION
[0033] To make the purpose, technical scheme and advantages of the embodiments of the present application clearer, the technical scheme in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all of the embodiments. The components of the embodiments of the present application usually described and shown in the drawings here can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present application provided in the drawings is not intended to limit the scope of the application claimed for protection, but merely represents the selected embodiments of the present application. Based on the embodiments of the present application, each other embodiment obtained by those skilled in the art without making creative work belongs to the scope of protection of the present application.
[0034] It is worth noting that before the present application is proposed, the network has become an indispensable part of people's lives. However, with the continuous development of Internet technology, the frequency of network attacks is also increasing. These network attacks will not only cause a lot of trouble to users, but also endanger the security and stability of the network to a certain extent. In order to prevent the occurrence of network security incidents, network security detection plays a vital role in ensuring network security. Traditional technology usually compiles regular expressions into deterministic finite automatons (Deterministic finite automaton, DFA) to perform network security detection tasks, that is, the regular expression detection matching engine converts NFA into DFA, and then the DFA matches the network data (i.e., the text to be detected) to achieve network security detection. However, when the traditional technology uses the DFA scanning engine to detect the text, serial matching is performed through the state transition table, that is, according to the order of the characters in the text to be detected, the text detection is performed one character at a time, resulting in the problem of low text detection efficiency.
[0035] Based on this, an embodiment of the present application provides a text detection method to improve text detection efficiency.
[0036] See also Figure 1 , Figure 1 The following is a flow chart of a text detection method provided in an embodiment of the present application. Figure 1 As shown, the text detection method provided in the embodiment of the present application includes:
[0037] Step S101 , dividing the text to be detected into multiple text segments, so as to use multiple threads to perform text detection on different text segments at the same time.
[0038] In this step, the text to be detected can be understood as the object to be detected. The text to be detected is a large text and includes multiple characters. The multiple characters can be Chinese, English, or other special characters, such as operators, identifiers, etc.
[0039] The multiple text segments may refer to partial segments of the text to be detected. The multiple text segments together constitute the text to be detected. Each text segment includes multiple characters. For example, the text to be detected T is divided into T1, T2, T3, ..., T n There are n text segments, and the n text segments are obtained by segmenting according to the order of characters in the text to be detected T. For example, the first group of 10,000 characters of the text to be detected T corresponds to the T1 text segment, the second group of 10,000 characters of the text to be detected T corresponds to the T2 text segment, and so on.
[0040] In an embodiment of the present application, multi-threading technology is used to open up multiple threads, and the number of threads is the same as the number of text segments, so that one thread detects one text segment. Assuming that the text T to be detected is divided into 10 text segments, 10 threads can be used to perform parallel detection on these 10 text segments at the same time to improve the text detection efficiency.
[0041] Before using multiple threads to perform text detection on multiple text segments at the same time, it is necessary to establish a trusted character set and an untrusted character set. Here, first, according to the regular expression used when detecting text, a DFA state transition table is obtained, and which characters in the state transition table will cause the most frequently accessed state to transfer, the characters that cause the most frequently accessed state to transfer are called untrusted characters in the untrusted character set, and the set consisting of all untrusted characters is called the untrusted character set; the characters that do not cause the most frequently accessed state to transfer are called trusted characters in the trusted character set, and the set consisting of all trusted characters is called the trusted character set. Among them, the most frequently accessed state refers to the zero state (0 state or state 0). That is, after inputting a character, if the current state transfers from the zero state to other non-zero states, for example: from state 0 to state 3, then the character is regarded as an untrusted character, and the untrusted character is a character that can cause the zero state to transfer; if the current state transfers from the zero state to the zero state, that is, the zero state does not transfer, then the character is regarded as a trusted character.
[0042] Refer to the following Figure 2 Let's introduce trusted characters and untrusted characters.
[0043] Figure 2 A schematic diagram of a state transition table provided in an embodiment of the present application is shown. Figure 2As shown, the state transition table has 5 states, namely state 0, state 1, state 2, state 3, and state 4; there are 5 characters, namely a, b, c, d, and e. When the most frequently accessed state is state 0, if characters a, b, and d are detected, the obtained transition states are 3, 2, and 1 respectively (the current state changes from 0 to 3, 2, and 1). If characters c and e are detected, the obtained transition states are all 0 (the current state changes from 0 to 0). Therefore, in this example, characters a, b, and d are untrusted characters in the untrusted character set, and characters c and e are trusted characters in the trusted character set.
[0044] Step S102, for each text segment, when using the target thread to detect the text segment, multiple target characters in the text segment are obtained at one time, and whether there are untrusted characters in the multiple target characters is determined in parallel.
[0045] In this step, in order to improve the efficiency of text detection, multiple target characters can be acquired at one time, and parallel processing can be performed on the multiple target characters at the same time.
[0046] In the embodiment of the present application, the initial state of all text segments is set to state 0, that is, to the most frequently accessed state, and subsequent processing is performed under this premise. Since the processing of multiple text segments in different threads is the same, the following description will only take one thread detecting one text segment as an example.
[0047] Refer to the following Figure 3 To introduce the parallel processing of multiple target characters, Figure 3 A flow chart of a method for determining whether untrusted characters exist in a plurality of target characters provided by an embodiment of the present application is shown. Figure 3 As shown, step S102 includes:
[0048] Step S102A, obtaining multiple target characters at one time by loading character blocks.
[0049] Taking text segment A as an example, thread a corresponding to text segment A will obtain multiple target characters from text segment A at one time through the LoadCharBlock() character block loading function, that is, execute CharBlock=LoadCharBlock(). Among them, the character block loading function is used to read the characters at positions [pos, pos+1, ..., pos+W-1] in text segment A. Since the starting state of each text segment is 0, when the characters in text segment A are obtained for the first time, W target characters can be obtained at one time. W is the character width corresponding to the Intel AVX vector, for example: for a 512-bit wide system, W=64. Among them, pos represents the current character position, and pos=0 at the initial time.
[0050] Step S102B, using a vector comparison instruction to perform a parallel comparison of the multiple target characters with the untrusted character set to determine whether there is a target character in the multiple target characters that matches the untrusted character in the untrusted character set.
[0051] Execute MaskVec[i] = _mm512_cmpeq_epi8(CharBlock, UntrustCharSet[i]), where i∈[0,K-1], K represents the number of untrusted characters. _mm512_cmpeq_epi8() is a built-in vector comparison instruction. The _mm512_cmpeq_epi8() function has a fast data comparison function based on single instruction multiple data stream (SIMD instruction). It can compare 512-bit (64 characters) packed integer data with 64 untrusted characters a. By performing one comparison, it can determine which characters are equal to the untrusted character a. If the character is equal to the untrusted character a (8-bit integer), the corresponding bit in MaskVec is set to 0xFF (all bits are 1), otherwise it is set to 0x00. If there is no such function, it is necessary to process these 64 target characters one by one through a loop to compare with the untrusted character a, and 64 steps are required. 512 is the bit width corresponding to the Intel AVX vector. Depending on the system, this value can also be 128 or 256. In the future, with the development of hardware technology, it may be expanded to 1024 or even a larger bit width. UntrustCharSet represents the untrusted character set. Through this vector comparison instruction, the 64 target characters obtained in the CharBlock at one time can be compared with each untrusted character in the untrusted character set in parallel to determine whether there is an untrusted character in the 64 target characters. Then all MaskVec[i] constitute MaskVec.
[0052] Since the comparison result MaskVec is a bit vector, it is necessary to convert the bit vector into a mask. Here, Mask = _mm512_movemask_epi8_512(MaskVec) can be executed. Among them, 512 is the bit width corresponding to the Intel AVX vector, and the _mm512_movemask_epi8_XX() function is used to convert each byte in the vector MaskVec[i] into a corresponding mask, and each mask bit indicates whether the corresponding byte is non-zero. For example: if the MaskVec bit width is 512 bits, then the converted Mask is 64 bits, and each bit indicates that the byte corresponding to MaskVec is zero or non-zero.
[0053] Step S102C: if there is a matching target character, determine whether there is an untrusted character among the multiple target characters.
[0054] If Mask≠0, it means that there is an untrusted character among the 64 target characters, that is, after the character is input for detection, the current state changes from state 0 to other states except state 0, and step S103 is executed. If Mask=0, it means that there is no untrusted character among the 64 target characters, that is, the current state changes from state 0 to state 0, and there is no need to detect these 64 target characters.
[0055] If there are no untrusted characters, the text detection of the multiple target characters is skipped, and the multiple characters after the multiple target characters are selected as new multiple target characters to determine whether there are untrusted characters in the new multiple target characters. Here, after skipping the detection of these 64 (0th to 63rd) target characters, the 64 characters from the 64th to the 127th in the text segment are obtained at one time as the multiple target characters of the second group to determine whether there are untrusted characters in the new multiple target characters. By analogy, steps S102A to S102C are continuously repeated to determine whether there are untrusted characters in each subsequent group of multiple target characters, until it is determined that there are untrusted characters in a certain group of multiple target characters in the text segment, and then step S103 is executed.
[0056] Step S103: if there is an untrusted character, then for each target character after the first untrusted character among the multiple target characters, determine the transfer state corresponding to the target character.
[0057] In this step, the transfer state may refer to the latest current state obtained after detecting the target character. For example, if the current state is 0 and the current state changes from 0 to 3 after detecting the target character a, then state 3 is the latest current state, that is, the transfer state corresponding to the target character a.
[0058] Assume that multiple target characters are b, d, and c. Since there are untrusted characters, the position of the first untrusted character b is obtained, and the state transition table is used to determine the transfer state corresponding to each target character after the untrusted character b, wherein the target characters after the untrusted character b include the first untrusted character b. Here, the position of the first untrusted character can be obtained by the pos+=__builtin_ctz(Mask) function, wherein the __builtin_ctz() function is used to return the number of bits from the highest bit (i.e., the rightmost bit) to the first bit that is 1 in the binary representation of the input integer Mask. For example, for Mask=8 (binary representation is 1000), the value returned by __builtin_ctz(Mask) is 3, because there are 3 bits from the highest bit to the first bit that is 1. Then, the state transition table in the memory is accessed, and the state transition table is used to determine the transfer state corresponding to each target character.
[0059] It should be noted that, since the transfer state of the trusted character is state 0, and the acceptable state is not state 0, the trusted character can be skipped directly, and the position of the first untrusted character can be directly obtained to determine the transfer state of each target character starting from this position, which can improve the text detection efficiency.
[0060] Refer to the following Figure 4 Let's introduce the text detection process when the transfer state is not the target state. Figure 4 FIG. 4 shows a flowchart of a text detection method when the transition state is not the target state provided by an embodiment of the present application. Figure 4 As shown, after step S103, the following further includes:
[0061] Step S103A: if the transfer state is not the target state, determine whether the current character is outside the preset range.
[0062] Here, the current character is the next character of the character, and the target state may refer to the zero state (state 0). The transfer state corresponding to the first untrusted character b is obtained through the state transfer table. Since the transfer state corresponding to the untrusted character b is not state 0, the next character of the untrusted character b is taken as the current character, that is, the target character d is taken as the current character, and it is determined whether the position of the current character d is outside the preset range. Among them, the preset range refers to [0, DANGER_AREA], and DANGER_AREA is a fixed value, for example: 100. Those skilled in the art can select the specific value of DANGER_AREA according to actual conditions, and this application does not limit it here.
[0063] Step S103B: if the current character is outside the preset range, determine whether there is an acceptable state in the transition state corresponding to the current character.
[0064] If the current character d is within the preset range [0,100], the next character of the target character d is taken as the new current character, that is, the target character c is taken as the new current character, and the process returns to the step of determining the transition state corresponding to the target character, that is, determining the transition state corresponding to the current character c. Steps S103A to S103C are continued for the current character c, and so on, as long as the transition state corresponding to each target character is not state 0.
[0065] If the current character d is outside the preset range [0,100], that is, the current character d is after the 101st character of the text segment, then determine whether there is an acceptable state in the transition state corresponding to the current character d. Taking the acceptable state as 4 as an example, determine whether the transition state corresponding to the 101st character is 4.
[0066] If it is determined that the transition state corresponding to the 101st character is 4, it means that the text segment may hit the regular expression rule. If there is no acceptable state, the next character of the current character d is continued as the new current character and multiple new target characters, and steps S103A to S103C are continued for the current character, and so on. As long as the transition state corresponding to each target character is not state 0, the above process is repeated to continue text detection for the new current character.
[0067] Step S104, if the transfer state is the target state, return to execute the step of acquiring multiple target characters in the text segment at one time, so as to perform text detection on the newly acquired multiple target characters.
[0068] In this step, if the transfer state corresponding to the current character is state 0, it means that the transfer state at this time is the same as the starting state of the text segment. At this time, the single instruction multiple data stream can continue to be used to process multiple target characters in parallel, that is, repeat the above steps S102 to S104 to continue text detection for the next group of multiple target characters. Among them, the single instruction multiple data stream can simultaneously carry out multiple processors through a controller, and each group of data can be independently operated and executed, which truly realizes the parallel processing of data.
[0069] Repeating the above steps, the transition states corresponding to all characters in each text segment can be determined (the transition state corresponding to the trusted character is state 0). However, since the starting state of each text segment is preset and may not be accurate, it is necessary to determine whether the preset starting state of each text segment is correct in turn according to the order of the text segments. If the starting state is incorrect, the text detection needs to be repeated.
[0070] Refer to the following Figure 5 Let's introduce the status detection process. Figure 5 A flow chart of the state detection method provided in the embodiment of the present application is shown as follows: Figure 5 As shown, after step S104, the following steps are also included:
[0071] Step S105, determining whether the ending state of the first text segment is equal to the preset starting state of the second text segment.
[0072] Here, the first text segment and the second text segment are two adjacent text segments among multiple text segments. For example, if the text to be detected is divided into ten text segments, if the first text segment is the second text segment, then the second text segment is the third text segment; if the first text segment is the third text segment, then the second text segment is the fourth text segment.
[0073] It should be noted that when the state detection of the text segment is performed, it starts from the first text segment and the second text segment, that is, the first text segment of the first round of state detection refers to the first text segment, and the second text segment refers to the second text segment. After the state detection of the first text segment and the second text segment is completed, the second round of state detection begins, and in this round, the second text segment is used as the first text segment, the third text segment is used as the second text segment, and so on, until the last round of state detection, the second to last text segment is used as the first text segment, and the last text segment is used as the second text segment.
[0074] Specifically, taking the current state detection round as the first round of state detection as an example, the first text segment is the first text segment, and the second text segment is the second text segment. The transition state corresponding to the last character in the first text segment is used as the end state of the first text segment, and it is determined whether the end state of the first text segment is equal to the preset starting state of the second text segment, that is, whether the end state of the first text segment is equal to 0.
[0075] Step S106: If the ending state is equal to the starting state, determine whether the text to be detected hits the rule based on the multiple transition states corresponding to the first text segment and the second text segment.
[0076] If the ending state of the first text segment is equal to 0, determine whether there is an acceptable state among the multiple transition states corresponding to the first text segment and the second text segment. If there is an acceptable state among the multiple transition states corresponding to the first text segment and the second text segment, determine that the text to be detected hits the rule, that is, the first text segment and the second text segment hit the regular expression rule.
[0077] If there is no acceptable state in the multiple transition states corresponding to the first text segment and the second text segment, the second text segment is used as a new first text segment, and the text segment after the second text segment is used as a new second text segment, so as to determine whether the text to be detected hits the rule according to the new first text segment and the new second text segment. That is, the second text segment is used as a new first text segment, and the third text segment is used as a new second text segment, and the execution starts again from step S105 to start a new round of state detection to determine whether the second text segment and the third text segment hit the regular expression rule, and so on, until the last round of state detection is completed to determine which text segments among all text segments hit the regular expression rule.
[0078] Step S107 , if the ending state is not equal to the starting state, the second text segment is detected by the first detection path and the second detection path respectively to obtain a plurality of first path transition states and a plurality of second path transition states.
[0079] The first detection path is a path that uses the ending state of the first text segment as the starting state of the second text segment to detect characters within a preset range in the second text segment; the second detection path is a path that detects characters within a preset range in the second text segment while keeping the starting state of the second text segment unchanged.
[0080] The characters in [0, DANGER_AREA] in the second text segment are detected by the first detection path to obtain a plurality of first path transition states. The characters in [0, DANGER_AREA] in the second text segment are detected by the second detection path to obtain a plurality of second path transition states.
[0081] Step S108, determining whether the plurality of first path transfer states intersect with the plurality of second path transfer states.
[0082] Specifically, for each first path transfer state, it is determined whether the second path transfer state at the corresponding position is the same as the first path transfer state; if they are the same, it is determined that multiple first path transfer states intersect with multiple first path transfer states.
[0083] For example: the multiple first path transfer states are 2, 1, 3, 5, 2, 0, 1, 3, and the multiple second path transfer states are 1, 2, 3, 5, 2, 0, 1, 3. Since the transfer state at the third position in the multiple first path transfer states and the multiple second path transfer states is 3, that is, the second path transfer state at the corresponding position is the same as the first path transfer state, it is determined that there is an intersection between the multiple first path transfer states and the multiple first path transfer states. On the contrary, if each first path transfer state is different from the second path transfer state at the corresponding position, it is determined that there is no intersection between the multiple first path transfer states and the multiple first path transfer states. Since the first path transfer state and the second path transfer state are transfer states corresponding to the same group of characters, if there is an intersection between the two path transfer states, the transfer states after the intersection must be all the same.
[0084] Step S109: If there is an intersection and an acceptable state exists during the detection process, the hit rule of the text to be detected is determined.
[0085] Here, the detection process includes a state detection process and a detection process of confirming the transfer state of the target character in step S103, wherein the detection process of confirming the transfer state of the target character in step S103 includes the first text segment and the second text segment. Taking the acceptable state as state 4 as an example, in the case of intersection, if there is a transfer state equal to 4 in the detection process, it can be determined that the second text segment hits the regular expression rule.
[0086] If there is an intersection and there is no acceptable state during the detection process, it means that neither the first text segment nor the second text segment hits the regular expression rule, and it is necessary to continue to detect the subsequent text segments, then the second text segment is used as the new first text segment, and the text segment after the second text segment is used as the new second text segment, so as to determine whether the text to be detected hits the rule based on the new first text segment and the new second text segment. For example: if the second text segment is the second text segment, then the second text segment is used as the new first text segment, and the third text segment is used as the new second text segment, and the execution of step S105 is returned, and the cycle is repeated until the status detection of all text segments is completed.
[0087] Step S110, if there is no intersection and there is no acceptable state in the detection process for the first text segment, the end state of the first text segment is used as the start state of the second text segment, and text detection is performed on the characters outside the preset range in the second text segment according to the start state.
[0088] If there is no intersection and there is no acceptable state in the detection process for the first text segment, it means that the results obtained by performing text detection on the second text segment according to the end state of the first text segment and the text detection on the second text segment according to the set start state of the second text segment are quite different, and the second text segment needs to be re-detected. Therefore, the end state of the first text segment is used as the start state of the second text segment, and the characters in the interval [DANGER_AREA, N] in the second text segment are re-detected according to the start state. Among them, the characters in the interval [DANGER_AREA, N] are characters outside the preset range, and N represents the position of the last character of the second segment. For example: if the second text segment has a total of 10001 characters, then [100, 10000]. The reason why the characters in the interval [DANGER_AREA, N] in the second text segment are directly detected is that [0, DANGER_AREA] in the second text segment has been detected by the first detection path before, and multiple first path transfer states have been obtained.
[0089] After completing the re-detection of the second text segment, this round of status detection process ends, and the second text segment is used as the new first text segment, and the next text segment of the second text segment is used as the new second text segment to start the next round of status detection for the new first text segment and the new second text segment, and so on, until the status detection of all text segments is completed.
[0090] Compared with the text detection method in the traditional technology, the present application can use multi-threading to simultaneously detect multiple text segments of the text to be detected, avoiding the problem of slow text detection caused by serial detection of each character in turn. At the same time, it can also perform parallel detection on multiple target characters read at one time, so as to accelerate parallel detection at both the text segment and character levels, further improving the text detection efficiency and solving the problem of low text detection efficiency in the traditional technology.
[0091] Based on the same inventive concept, a text detection device corresponding to the text detection method is also provided in the embodiment of the present application. Since the principle of solving the problem by the device in the embodiment of the present application is similar to the above-mentioned text detection method in the embodiment of the present application, the implementation of the device can refer to the implementation of the method, and the repeated parts will not be repeated.
[0092] See also Figure 6 , Figure 6 This is a schematic diagram of the structure of a text detection device provided in an embodiment of the present application. Figure 6 As shown in , the text detection device 200 includes:
[0093] The text segmentation module 201 is used to segment the text to be detected into multiple text segments, so as to use multiple threads to perform text detection on different text segments at the same time;
[0094] The untrusted character detection module 202 is used for obtaining multiple target characters in each text segment at one time when detecting the text segment using the target thread, and determining in parallel whether there are untrusted characters in the multiple target characters, wherein the untrusted characters are characters that can cause the zero state to transfer;
[0095] The transfer state determination module 203 is used to determine the transfer state corresponding to each target character after the first untrusted character among the multiple target characters if there is an untrusted character;
[0096] The loop iteration module 204 is used to return to the step of acquiring multiple target characters in the text segment at one time if the transfer state is the target state, so as to perform text detection on the newly acquired multiple target characters.
[0097] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.
[0098] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. The device embodiments described above are merely schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some communication interfaces, and the indirect coupling or communication connection of devices or units can be electrical, mechanical or other forms.
[0099] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.
[0100] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.
[0101] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a non-volatile computer-readable storage medium that is executable by a processor. Based on this understanding, the technical solution of the present application can be essentially or partly embodied in the form of a software product that contributes to the traditional technology. The computer software product is stored in a storage medium, including several instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.
[0102] Finally, it should be noted that the above-described embodiments are only specific implementation methods of the present application, which are used to illustrate the technical solutions of the present application, rather than to limit them. The protection scope of the present application is not limited thereto. Although the present application is described in detail with reference to the above-mentioned embodiments, ordinary technicians in the field should understand that any technician familiar with the technical field can still modify the technical solutions recorded in the above-mentioned embodiments within the technical scope disclosed in the present application, or can easily think of changes, or make equivalent replacements for some of the technical features therein; and these modifications, changes or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application, and should be included in the protection scope of the present application. Therefore, the protection scope of the present application shall be based on the protection scope of the claims.
Claims
1. A text detection method, characterized in that: include: Divide the text to be detected into multiple text segments, so as to use multiple threads to perform text detection on different text segments at the same time; For each text segment, when using the target thread to detect the text segment, multiple target characters in the text segment are obtained at one time, and it is determined in parallel whether there are untrusted characters among the multiple target characters, wherein the untrusted characters are characters that can cause the zero state to transfer; If there is an untrusted character, determining a transfer state corresponding to each target character after the first untrusted character among the multiple target characters; If the transfer state is the target state, returning to the step of acquiring multiple target characters in the text segment at one time to perform text detection on the newly acquired multiple target characters; determining whether an end state of a first text segment is equal to a preset start state of a second text segment, wherein the first text segment and the second text segment are two text segments that are adjacent to each other in the plurality of text segments; If the end state is not equal to the start state, the second text segment is detected by a first detection path and a second detection path respectively to obtain a plurality of first path transition states and a plurality of second path transition states, wherein the first detection path is a path for detecting characters within a preset range in the second text segment by taking the end state of the first text segment as the start state of the second text segment, and the second detection path is a path for detecting characters within a preset range in the second text segment when the start state of the second text segment remains unchanged; determining whether the plurality of first path transfer states intersect with the plurality of second path transfer states; If there is an intersection and an acceptable state exists during the detection process, determining the hit rule of the text to be detected; If there is no intersection and there is no acceptable state during the detection process of the first text segment, the end state of the first text segment is used as the starting state of the second text segment, and text detection is performed on characters outside a preset range in the second text segment based on the starting state.
2. The method according to claim 1, characterized in that After determining the transfer state corresponding to the target character, the method further includes: If the transition state is not the target state, determining whether the current character is outside a preset range, the current character is the next character of the character; If the current character is outside the preset range, it is determined whether there is an acceptable state in the transition state corresponding to the current character.
3. The method according to claim 1, characterized in that The step of acquiring multiple target characters in the text segment at one time and determining whether untrusted characters exist in the multiple target characters in parallel includes: Get multiple target characters at once by loading character blocks; Comparing the plurality of target characters in parallel with the untrusted character set using a vector comparison instruction to determine whether there is a target character in the plurality of target characters that matches an untrusted character in the untrusted character set; If there is a matching target character, it is determined that there is an untrusted character in the multiple target characters.
4. The method according to claim 1, characterized in that: The method further comprises: If the ending state is equal to the starting state, it is determined whether the text to be detected hits the rule according to a plurality of transition states corresponding to the first text segment and the second text segment.
5. The method according to claim 4, characterized in that The determining whether the plurality of first path transfer states intersect with the plurality of second path transfer states comprises: For each first path transfer state, determining whether a second path transfer state at a corresponding position is the same as the first path transfer state; If they are the same, it is determined that the plurality of first path transfer states intersect with the plurality of first path transfer states.
6. The method according to claim 1, characterized in that After the parallel determination of whether there are untrusted characters in the plurality of target characters, the method further includes: If there is no untrusted character, the text detection of the multiple target characters is skipped, and multiple characters after the multiple target characters are selected as new multiple target characters to determine whether there is an untrusted character in the new multiple target characters.
7. The method according to claim 4, characterized in that The step of determining whether the text to be detected hits a rule according to a plurality of transition states corresponding to the first text segment and the second text segment includes: If there is an acceptable state among the multiple transition states corresponding to the first text segment and the second text segment, determining that the to-be-detected text hits the rule; If there is no acceptable state among the multiple transition states corresponding to the first text segment and the second text segment, the second text segment is used as a new first text segment, and the text segment after the second text segment is used as a new second text segment, so as to determine whether the text to be detected hits the rule based on the new first text segment and the new second text segment.
8. The method according to claim 1, characterized in that After the if there is an intersection and an acceptable state exists in the detection process, it also includes: If there is an intersection and there is no acceptable state during the detection process, the second text segment is used as a new first text segment, and the text segment after the second text segment is used as a new second text segment, so as to determine whether the text to be detected hits the rule based on the new first text segment and the new second text segment.
9. A text detection device, characterized in that: include: A text segmentation module is used to divide the text to be detected into multiple text segments, so as to use multiple threads to perform text detection on different text segments at the same time; An untrusted character detection module is used for obtaining multiple target characters in each text segment at one time when detecting the text segment using a target thread, and determining in parallel whether there are untrusted characters among the multiple target characters, wherein the untrusted characters are characters that can cause a zero state to be transferred; A transfer state determination module, configured to determine, if there is an untrusted character, for each target character after the first untrusted character among the plurality of target characters, a transfer state corresponding to the target character; A loop iteration module, for returning to the step of acquiring multiple target characters in the text segment at one time if the transfer state is the target state, so as to perform text detection on the newly acquired multiple target characters; The loop iteration module is further used to determine whether the end state of the first text segment is equal to the preset start state of the second text segment, and the first text segment and the second text segment are two text segments adjacent to each other in the plurality of text segments; If the end state is not equal to the start state, the second text segment is detected by a first detection path and a second detection path respectively to obtain a plurality of first path transition states and a plurality of second path transition states, wherein the first detection path is a path for detecting characters within a preset range in the second text segment by taking the end state of the first text segment as the start state of the second text segment, and the second detection path is a path for detecting characters within a preset range in the second text segment when the start state of the second text segment remains unchanged; determining whether the plurality of first path transfer states intersect with the plurality of second path transfer states; If there is an intersection and an acceptable state exists during the detection process, determining the hit rule of the text to be detected; If there is no intersection and there is no acceptable state during the detection process of the first text segment, the end state of the first text segment is used as the starting state of the second text segment, and text detection is performed on characters outside a preset range in the second text segment based on the starting state.
