Message data processing method and device, computer equipment and storage medium

Abstract

The invention relates to a message data processing method and device, computer equipment, a storage medium and a computer program product. The method comprises the following steps: acquiring message data, and analyzing the message data into a plurality of key value pairs in a text form; aiming at each key value pair, determining a coding language type matched with the key value pair, segmenting a value in the key value pair into a plurality of text segments, mapping each text segment into a preset text mark based on the coding language type, and obtaining a text mark sequence; encoding the text mark sequence to obtain text mark features of values in the key value pairs; identifying a text category to which a value in the key value pair belongs according to the text mark feature; and determining the category of the message data according to the text categories to which the values of the plurality of key value pairs belong. By adopting the method, the message data category detection accuracy can be improved.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a message data processing method, apparatus, computer equipment, storage medium, and computer program product. Background Art

[0002] With the advancement of computer technology, network applications are becoming increasingly diverse, and network coverage is expanding. Network security is becoming increasingly important in production and daily life. At the same time, security incidents such as web page defacement and SQL (Structured Query Language) injection are becoming more frequent. Network access presents numerous security issues, requiring appropriate protection.

[0003] Traditionally, network protection has primarily been achieved through category detection of packet data transmitted across the network. This involves extracting information from the packet data and performing rule matching. Regular expressions are commonly used as matching patterns to describe the string or character sequence to be searched or replaced.

[0004] However, in traditional methods, when regular expression rules are used as matching patterns, the regular expression rules usually need to be pre-configured. If more complex rules are expressed, more complex regular expressions are required, which is prone to errors and misdetection, resulting in low category detection accuracy. Summary of the Invention

[0005] Based on this, it is necessary to provide a message data processing method, apparatus, computer equipment, computer-readable storage medium and computer program product that can improve the accuracy of message data category detection in response to the above technical problems.

[0006] In a first aspect, the present application provides a method for processing message data. The method comprises:

[0007] Obtaining message data, and parsing the message data into multiple key-value pairs in text form;

[0008] For each key-value pair, determining a coding language type that matches the key-value pair, dividing the value in the key-value pair into a plurality of text segments, and mapping each of the text segments to a preset text tag based on the coding language type to obtain a text tag sequence;

[0009] Encoding the text mark sequence to obtain text mark features of the values in the key-value pairs;

[0010] Identifying the text category to which the value in the key-value pair belongs based on the text mark feature;

[0011] The category of the message data is determined according to the text category to which the values of each of the multiple key-value pairs belong.

[0012] In a second aspect, the present application further provides a message data processing device. The device comprises:

[0013] A message data acquisition module, used to acquire message data and parse the message data into multiple key-value pairs in text form;

[0014] a text tag generation module, configured to determine, for each key-value pair, a coding language type that matches the key-value pair, and segment the value in the key-value pair into a plurality of text segments; and based on the coding language type, map each of the text segments to a preset text tag to obtain a text tag sequence;

[0015] An encoding module, configured to encode the text mark sequence to obtain text mark features of the values in the key-value pairs;

[0016] A category identification module, configured to identify the text category to which the value in the key-value pair belongs based on the text mark feature;

[0017] The processing module is used to determine the category of the message data according to the text category to which the values of each of the multiple key-value pairs belong.

[0018] In a third aspect, the present application further provides a computer device. The computer device includes a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the following steps are performed:

[0019] Obtaining message data, and parsing the message data into multiple key-value pairs in text form;

[0020] For each key-value pair, determining a coding language type that matches the key-value pair, dividing the value in the key-value pair into a plurality of text segments, and mapping each of the text segments to a preset text tag based on the coding language type to obtain a text tag sequence;

[0021] Encoding the text mark sequence to obtain text mark features of the values in the key-value pairs;

[0022] Identifying the text category to which the value in the key-value pair belongs based on the text mark feature;

[0023] The category of the message data is determined according to the text category to which the values of each of the multiple key-value pairs belong.

[0024] In a fourth aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the following steps:

[0025] Obtaining message data, and parsing the message data into multiple key-value pairs in text form;

[0026] For each key-value pair, determining a coding language type that matches the key-value pair, dividing the value in the key-value pair into a plurality of text segments, and mapping each of the text segments to a preset text tag based on the coding language type to obtain a text tag sequence;

[0027] Encoding the text mark sequence to obtain text mark features of the values in the key-value pairs;

[0028] Identifying the text category to which the value in the key-value pair belongs based on the text mark feature;

[0029] The category of the message data is determined according to the text category to which the values of each of the multiple key-value pairs belong.

[0030] In a fifth aspect, the present application further provides a computer program product. The computer program product includes a computer program that, when executed by a processor, implements the following steps:

[0031] Obtaining message data, and parsing the message data into multiple key-value pairs in text form;

[0032] For each key-value pair, determining a coding language type that matches the key-value pair, dividing the value in the key-value pair into a plurality of text segments, and mapping each of the text segments to a preset text tag based on the coding language type to obtain a text tag sequence;

[0033] Encoding the text mark sequence to obtain text mark features of the values in the key-value pairs;

[0034] Identifying the text category to which the value in the key-value pair belongs based on the text mark feature;

[0035] The category of the message data is determined according to the text category to which the values of each of the multiple key-value pairs belong.

[0036] The above-mentioned message data processing method, device, computer equipment, storage medium and computer program product, based on the acquisition of message data, can obtain multiple key-value pairs in the parsed text form by deeply parsing the message data. For each key-value pair, after determining the coding language type that matches the key-value pair and dividing the value in the key-value pair into multiple text fragments, each text fragment is mapped to a preset text flag with reference to the coding language type, so as to obtain a text flag sequence that can represent the value in the key-value pair. By encoding the text flag sequence, the features in the text flag sequence can be extracted to obtain the text flag features of the value in the key-value pair. Then, the text flag features can be used to accurately identify the text category to which the value in the key-value pair belongs. On this basis, according to the text category to which the values of each of the multiple key-value pairs belong, the category of the message data can be accurately determined. The entire process, based on parsing the message data and obtaining multiple key-value pairs in text form, can determine the text category to which the value in the key-value pair belongs through steps such as segmentation, mapping, encoding, and identification of text categories, and then use the text categories to which the values of multiple key-value pairs belong to determine the category of the message data. That is, the category of the message data is determined by analyzing the text content of the message data without the need to configure complex regular expression rules, which can improve the accuracy of message data category detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Figure 1 This is an application environment diagram of a message data processing method in one embodiment;

[0038] Figure 2 A diagram illustrating an application environment of a message data processing method in another embodiment;

[0039] Figure 3 1 is a flow chart of a method for processing message data in one embodiment;

[0040] Figure 4 Schematic diagram of a message format in one embodiment;

[0041] Figure 5 A schematic diagram of obtaining a text mark sequence in one embodiment;

[0042] Figure 6 A schematic diagram of a preset text mark and text mode in one embodiment;

[0043] Figure 7 A schematic diagram of determining a matching text mark in one embodiment;

[0044] Figure 8 A schematic diagram of obtaining a text mark sequence in another embodiment;

[0045] Figure 9 A schematic diagram of a process for identifying a text category to which a value in a key-value pair belongs in one embodiment;

[0046] Figure 10 A schematic diagram of determining a text category to which a value in a key-value pair belongs in one embodiment;

[0047] Figure 11 Schematic diagram of a flow chart of a message data processing method in another embodiment;

[0048] Figure 12 is a structural block diagram of a message data processing device in one embodiment;

[0049] Figure 13 FIG. 1 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0050] This application relates to the field of artificial intelligence technology. Artificial Intelligence (AI) is the theory, method, technology, and application system that uses digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use knowledge to achieve optimal results. In other words, AI is a comprehensive technology in computer science that attempts to understand the essence of intelligence and produce a new type of intelligent machine that can respond in a manner similar to human intelligence. AI is the study of the design principles and implementation methods of various intelligent machines, giving them the capabilities of perception, reasoning, and decision-making.

[0051] Artificial intelligence (AI) technology is a comprehensive discipline encompassing a wide range of fields, encompassing both hardware and software technologies. Foundational AI technologies generally include sensors, specialized AI chips, cloud computing, distributed storage, big data processing, pre-trained models, operating / interaction systems, and mechatronics. Pre-trained models, also known as large models or basic models, can be fine-tuned and widely applied to downstream tasks across various AI disciplines. AI software technologies primarily encompass computer vision, speech processing, natural language processing, and machine learning / deep learning.

[0052] Among them, the present application mainly involves natural language processing technology. Natural language processing (NLP) is an important direction in the fields of computer science and artificial intelligence. It studies various theories and methods that can enable effective communication between people and computers using natural language. Natural language processing involves natural language, that is, the language people use in daily life, and is closely related to linguistic research; it also involves computer science and mathematics. The pre-training model, an important technology for model training in the field of artificial intelligence, is developed from the large language model (Large Language Model) in the field of NLP. After fine-tuning, the large language model can be widely used in downstream tasks. Natural language processing technology generally includes text processing, semantic understanding, machine translation, robot question answering, knowledge graph and other technologies.

[0053] In order to make the purpose, technical solutions and advantages of this application more clear, the following further describes this application in detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain this application and are not intended to limit this application.

[0054] The message data processing method provided in the embodiment of the present application can be applied to Figure 1 In the application environment shown. Among them, the terminal 102 communicates with the server 104 through the network. The data storage system can store the data that the server 104 needs to process. The data storage system can be integrated on the server 104, or placed on the cloud or other servers. When the terminal 102 needs to request data from the server 104, it sends message data representing the request to the server 104. The server 104 obtains the message data, parses the message data into multiple key-value pairs in text form, determines the coding language type that matches the key-value pair for each key-value pair, and divides the value in the key-value pair into multiple text segments. Based on the coding language type, each text segment is mapped to a preset text marker, a text marker sequence is obtained, and the text marker sequence is encoded to obtain the text marker feature of the value in the key-value pair. Based on the text marker feature, the text category to which the value in the key-value pair belongs is identified. Based on the text category to which the values of the multiple key-value pairs belong, the category of the message data is determined. When the category of the message data is normal message data, the server 104 feedbacks the required data to the terminal 102 in response to the message data.

[0055] Among them, the terminal 102 can be, but is not limited to, various desktop computers, laptops, smart phones, tablet computers, Internet of Things devices, portable wearable devices and aircraft, etc. The Internet of Things devices can be smart speakers, smart TVs, smart air conditioners, smart car-mounted devices, etc. Portable wearable devices can be smart watches, smart bracelets, head-mounted devices, etc. The server 104 can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms. The terminal can be a smart phone, tablet computer, laptop computer, desktop computer, smart speaker, smart watch, etc., but is not limited to this. The terminal and the server can be directly or indirectly connected through wired or wireless communication, and this application does not limit this.

[0056] In one embodiment, the message data processing method provided in the embodiment of the present application can be applied to Figure 2 In the application environment shown, the terminal 202 communicates with a network protection server 204 for message data processing through a network, and the network protection server 204 communicates with a service server 206 for processing requests of the terminal 202 through a network. When terminal 202 needs to request data from service server 206, the message data sent by terminal 202 will first reach network protection server 204. Network protection server 204 obtains the message data and parses it into multiple key-value pairs in text form. For each key-value pair, it determines the encoding language type that matches the key-value pair and segments the value in the key-value pair into multiple text segments. Based on the encoding language type, each text segment is mapped to a preset text tag to obtain a text tag sequence. The text tag sequence is encoded to obtain text tag features of the value in the key-value pair. Based on the text tag features, the text category to which the value in the key-value pair belongs is identified. Based on the text categories to which the values of the multiple key-value pairs belong, the category of the message data is determined. If the message data category is normal message data, the message data is sent to service server 206, so that service server 206 can feedback the required data to terminal 102 in response to the message data. If the message data category is abnormal message data, the message data is blocked and not sent to service server 206. By first using the network protection server 204 to perform category detection on the message data, and only releasing the message data to the business server 206 for processing when it is determined that the category of the message data is normal message data, the business server 206 can be protected.

[0057] In one embodiment, Figure 3As shown, a message data processing method is provided, which can be executed by a terminal or a server alone, or by a terminal and a server in collaboration. In the embodiment of the present application, the method is applied to a server as an example for illustration, and includes the following steps:

[0058] Step 302: Acquire message data and parse the message data into multiple key-value pairs in text form.

[0059] Message data is the unit of data exchanged and transmitted within a network. It's a block of data that a station sends at once. During transmission, it's continuously encapsulated into packets, packets, and frames. Encapsulation involves adding information segments, which are the data organized in a specific format within the message header. For example, the information segments can include information such as the message type, version, length, and entity body. Message data contains the complete data to be sent, and its length varies widely, being unlimited and variable.

[0060] Here, the format of the message data is described by taking HTTP (Hypertext Transfer Protocol) message as an example. Message data usually has two message types: request and response. The message format can be as follows: Figure 4 As shown in the figure, they all consist of a start line, a message header, and a message body. The first line of the message data is called the start line. The start line is required and contains basic information about the request / response. The second line to the blank line of the message data is called the message header. Some message headers are optional, while others are required. It contains a sequence of key: value pairs (key is not case-sensitive) (e.g. Figure 4 The message body (shown as key1:value1, key2:value2, key3:value3, and so on) is used to describe additional information about the request or response. The message body, which begins with a blank line and continues until the last line, is called the message body. The message body is optional and carries the specific content to be uploaded or responded to. It may also include key-value pairs.

[0061] Key-value pairs are used to describe the request or message body in the message data. Specifically, they can be expressed in the form of key-value pairs (key:value), where the key can be the parameter name and the value can be the parameter value. That is, key-value pairs can appear in the message header and message body of the message data.

[0062] Specifically, after receiving the message data, the server determines the corresponding transmission protocol for the message data and uses the transmission protocol to perform fine-grained parsing of the message data, parsing the message data into multiple key-value pairs in text form. In specific applications, message data transmitted using different transmission protocols requires corresponding parsing methods. For example, if the HTTP protocol is used for transmission, the HTTP parsing method is used to parse the message data. If the WebSocket protocol (a protocol for full-duplex communication over a single TCP connection) is used for transmission, the WebSocket parsing method is used to parse the message data. The transmission protocol corresponding to the message data is the transmission protocol used when the message data is transmitted.

[0063] In a specific application, for example, where the transmission protocol is HTTP and the message data is an HTTP message, after receiving the HTTP message, the server will perform fine-grained HTTP protocol parsing (i.e., parsing using the parsing method specified for the HTTP protocol), parsing the message data into multiple key-value pairs in text form. In a specific application, fine-grained HTTP protocol parsing can include parsing URL (uniform resource locator) parameters, headers, cookies (small text files that are data (usually encrypted) stored on the local terminal of the user to identify the user, temporarily or permanently stored by the local terminal), JSON (JavaScript Object Notation, a lightweight data exchange format), and XML (eXtensible Markup Language) parsing. Through these parsing steps, multiple key-value pairs in text form can be obtained.

[0064] Step 304: for each key-value pair, determine the coding language type that the key-value pair matches, and divide the value in the key-value pair into multiple text segments. Based on the coding language type, map each text segment to a preset text tag to obtain a text tag sequence.

[0065] The coding language type that matches the key-value pair refers to the coding language type used by the key-value pair. This can be determined by identifying the identifier or statement structure in the key-value pair that represents the coding language type. For example, the coding language type can be SQL (Structured Query Language), JavaScript (a lightweight, interpreted or just-in-time compiled programming language with function-first semantics), or Shell (a programming language).

[0066] It is understandable that the identifier can specifically be a word that appears only in the coding language type. For example, the identifier can specifically be a string that appears only in the coding language type. For example, the SELECT (query) statement typically appears only in the Structured Query Language, so "SELECT" can be used as the identifier for the Structured Query Language. That is, when the identifier "SELECT" appears in a key-value pair, it can be determined that the coding language type that the key-value pair matches is the Structured Query Language. For another example, the FROM (source) statement typically appears only in the Structured Query Language, so "FROM" can be used as the identifier for the Structured Query Language. That is, when the identifier "FROM" appears in a key-value pair, it can be determined that the coding language type that the key-value pair matches is the Structured Query Language. The statement structure refers to the statement form specific to the coding language type. For example, the statement structure can specifically be a specific first character string or a specific unsigned character string that appears only in statements of the programming language type.

[0067] A key-value pair consists of two parts: the key and the value. The value in a key-value pair refers to the text portion of the value. For example, if the key-value pair is in the form of (key text: value text), the value in the key-value pair is the value text. For example, if the key-value pair is in the form of (key string: value string), the value in the key-value pair is the value string.

[0068] A text fragment refers to a fragment consisting of a portion of a key-value pair's value after it is segmented. For example, if the value in the key-value pair is a string, a text fragment refers to a fragment consisting of a portion of the string after the value in the key-value pair is segmented. It is understood that a fragment consisting of a portion of a string is also a string.

[0069] Among them, the preset text flag refers to a pre-set flag for representing the grammar of the text fragment, which can be configured according to the actual application scenario. It is understandable that each preset text flag is used to represent a grammar in the programming language type, that is, for each programming language type, multiple preset text flags are configured to represent the multiple grammars therein. It is understandable that there is a one-to-one correspondence between the multiple preset text flags and the multiple grammars in the programming language type. The text flag sequence refers to a sequence of text flags used to represent the value in the key-value pair, at least including each text fragment mapped to the text fragment, that is, the text flag sequence can be used to represent the value in the key-value pair.

[0070] Among them, a variety of preset text markers are configured according to the language characteristics of the programming language type. For example, the preset text marker can be specifically used to represent specific programming characters in the programming language type. Specific programming characters refer to special programming characters that only appear in the programming language type. For example, when the programming language type is SQL language, a preset text marker can be set for the specific programming characters "SELECT" and "FROM". For another example, the preset text marker can be specifically used to represent integers, names, strings and other grammatical forms in the programming language type. For another example, the preset text marker can also be used to represent grammatical symbols in the programming language type. For example, the grammatical symbols can be specifically "=", "OR", "<<", ":=", "and", etc.

[0071] Specifically, the server will obtain the language type identification feature set configured for multiple coding language types, and for each key-value pair, perform feature matching on the language type identification feature set of multiple coding language types with the key-value pair, obtain the matching identification feature that matches the key-value pair, and use the coding language type indicated by the matching identification feature as the coding language type that matches the key-value pair. In addition to determining the coding language type that matches the key-value pair, such as Figure 5 As shown, the server will also split the value in the key-value pair into multiple text fragments according to the preset delimiter (such as Figure 5 As shown, including text segment 1, text segment 2, text segment 3 and text segment 4), based on the coding language type, a text pattern of multiple preset text tags configured for the coding language type can be obtained, and according to the text patterns of the multiple preset text tags, each text segment is mapped to a preset text tag to obtain a text tag sequence.

[0072] In specific applications, each language type identification feature in the language type identification feature set can be configured according to the actual application scenario. For example, the language type identification feature can specifically be an identifier that characterizes the type of coded language. For another example, the language type identification feature can also be a structural regular expression that characterizes the sentence structure of the language coding type. It can be understood that when the language type identification feature is an identifier, as long as the key-value pair hits the identifier, the coding language type of the key-value pair can be determined. When the language category identification feature is a structural regular expression that characterizes the sentence structure of the language coding type, as long as the sentence structure of the key-value pair is the same as the structural regular expression, the coding language type of the key-value pair can be determined. The preset delimiter can be configured according to the actual application scenario. For example, the preset delimiter can specifically be a space, and the value in the key-value pair can be divided into multiple text segments according to the space.

[0073] In a specific application, this step is explained by taking the string "select 1 from admin wherems = 'test' or '1' = '1'" as an example. The server performs feature matching on the string using sets of identifiers of multiple coding language types (the identifier set of the structured query language includes the identifier "SELECT"), and can obtain the matching identification feature that matches the key-value pair as the identifier "SELECT". The structured query language indicated by the identifier "SELECT" can then be used as the coding language type that matches the key-value pair. At the same time, the server can split the string into multiple text segments (i.e., 12 text segments: "select", "1", "from", "admin", "where", "ms", "=", "'test'", "or", "'1'", "=", "1'") according to the preset delimiter (space).

[0074] Step 306: Encode the text tag sequence to obtain the text tag features of the values in the key-value pairs.

[0075] The text mark feature refers to a feature used to represent the value in the key-value pair. For example, the text mark feature can be a feature vector used to represent the value in the key-value pair.

[0076] Specifically, the server encodes the text token sequence to obtain the text token features of the value in the key-value pair. In specific applications, the server encodes the text token sequence to obtain an encoded feature sequence corresponding to the text token sequence. After obtaining the encoded feature sequence, the server can directly use the encoded feature sequence as the text token features of the value in the key-value pair, or process the encoded feature sequence based on the attention mechanism and use the features obtained after processing as the text token features of the value in the key-value pair.

[0077] In a specific application, a text marker sequence includes multiple text marker elements, and each text marker element is used to represent a text fragment. The text marker sequence is encoded, mainly encoding each text marker element therein. By encoding each text marker element in the text marker sequence separately, the respective encoding features of each text marker element can be obtained. The respective encoding features of each text marker element form a coding feature sequence corresponding to the text marker sequence.

[0078] In a specific application, the server can use a preset coding table to encode the text marker element and obtain the coding features of the text marker element, that is, the server can use the preset coding table to map the text marker element to the coding features. Among them, the preset coding table can be configured according to the actual application scenario. For example, the preset coding table can be specifically a digital coding table, and a plurality of digital codes are configured in the preset coding table, which can be used to represent different text marker elements. Taking the coding language type as structured query language as an example, the preset coding table can be as shown in Table 1, wherein a digital number is configured for the preset text marker in each text marker element. It can be understood that Table 1 only shows a part of the mapping relationship between the digital number and the preset text marker.

[0079] Table 1

[0080]

[0081] In a specific application, when processing a coded feature sequence based on an attention mechanism, the server can input the coded feature sequence into a pre-trained attention model, use the attention model to perform semantic understanding on the coded feature sequence, and obtain textual signature features of the values in the key-value pairs. In a specific application, the pre-trained attention model can be trained according to the actual application scenario, and this is not specifically limited in this embodiment.

[0082] Step 308: Identify the text category to which the value in the key-value pair belongs based on the text mark feature.

[0083] The text category is one of the preset text categories. It is understood that the preset text category refers to a pre-set possible text category and can be configured according to the actual application scenario. For example, in this embodiment, the preset text category can specifically be normal text and abnormal text. Abnormal text refers to text that is characterized as having security risks, while normal text refers to text that is characterized as not having security risks.

[0084] Specifically, the server will predict the text category based on the text marker features, obtain the probability that the value in the key-value pair belongs to each preset text category, and determine the text category to which the value in the key-value pair belongs based on the probability that the value in the key-value pair belongs to each preset text category. In a specific application, the server can determine the candidate text category with the highest probability from the probabilities that the value in the key-value pair belongs to each preset text category. When the probability that the value in the key-value pair belongs to the candidate text category is greater than the preset probability threshold, the candidate text category is used as the text category to which the value in the key-value pair belongs. The preset probability threshold can be configured according to the actual application scenario. For example, the preset probability threshold can be specifically 80%.

[0085] In a specific application, the server can input the text signature features into a pre-trained text classification model. The pre-trained text classification model can then identify the text category to which the value in the key-value pair belongs, and the text category output by the text classification model is used as the text category to which the value in the key-value pair belongs. The pre-trained text classification model can be trained according to actual application scenarios, and this is not specifically limited in this embodiment.

[0086] Step 310: Determine the category of the message data according to the text category to which the values of each of the multiple key-value pairs belong.

[0087] Specifically, the server can determine the category of the message data based on the text categories to which the values of each of the multiple key-value pairs belong. When the text categories to which the values of each of the multiple key-value pairs belong are all normal texts characterized as having no security risks, the category of the message data is a normal message. When the text category to which the value of any key-value pair belongs is abnormal text characterized as having security risks, the category of the message data is an abnormal message. It can be understood that an abnormal message is a message with security risks. In this embodiment, the presence of security risks means that it may have an adverse effect on the server that processes the message data, affecting the normal operation of the server that processes the message data.

[0088] The above-mentioned message data processing method, based on obtaining message data, can obtain multiple key-value pairs in the form of parsed text by deeply parsing the message data. For each key-value pair, after determining the coding language type that matches the key-value pair and dividing the value in the key-value pair into multiple text fragments, each text fragment is mapped to a preset text flag with reference to the coding language type, so as to obtain a text flag sequence that can represent the value in the key-value pair. By encoding the text flag sequence, the features in the text flag sequence can be extracted to obtain the text flag features of the value in the key-value pair. Then, the text flag features can be used to accurately identify the text category to which the value in the key-value pair belongs. On this basis, according to the text category to which the values of each of the multiple key-value pairs belong, the category of the message data can be accurately determined. The entire process, based on parsing the message data and obtaining multiple key-value pairs in text form, can determine the text category to which the value in the key-value pair belongs through steps such as segmentation, mapping, encoding, and identification of text categories, and then use the text categories to which the values of multiple key-value pairs belong to determine the category of the message data. That is, the category of the message data is determined by analyzing the text content of the message data without the need to configure complex regular expression rules, which can improve the accuracy of message data category detection.

[0089] In one embodiment, mapping each text segment to a preset text marker based on the encoding language type, and obtaining a text marker sequence includes:

[0090] Get the text mode of various preset text flags configured for the encoding language type;

[0091] For each text segment, pattern matching is performed on the text segments with the text patterns of the plurality of preset text markers to obtain matching text markers that match the text segment;

[0092] A text token sequence is obtained according to the matching text tokens of each text segment.

[0093] Among them, the text pattern refers to the form of a preset text marker, which is used to define the preset text marker, that is, the form that the preset text marker must satisfy. For example, the text pattern can specifically be a pattern string used to define the preset text marker, and the preset text marker must satisfy this string. For example, the preset text marker can be specifically used to represent a specific programming character in the programming language type, and the text pattern of this preset text marker can specifically be the specific programming character itself. For another example, the preset text marker can be specifically used to represent the grammatical forms such as integers, names, and strings in the programming language type. Then corresponding pattern strings can be configured for grammatical forms such as integers, names, and strings. For example, the pattern string corresponding to the grammatical form of integers can be any combination of any numbers. For another example, the pattern string corresponding to the grammatical form of names can be any combination of any letters + any letters and any numbers.

[0094] Taking the programming language type as structured query language as an example, the multiple preset text tags configured for the structured query language and the text patterns of the multiple preset text tags can be as follows: Figure 6 As shown. Among them, the multiple preset text tags include preset text tags representing specific programming characters "SELECT" and "FROM", preset text tags representing grammatical forms such as integers, names, and strings in programming language types, and preset text tags representing grammatical symbols such as "=", "OR", "<<", ":=", "and", "where", etc. A text mode is configured for each text tag, wherein the text modes of the preset text tags "SELECT" and "FROM" are the specific programming characters "SELECT" and "FROM" themselves, respectively. The text mode of the grammatical form of integers is any combination of any numbers (such as Figure 6 The pattern string corresponding to the name grammatical form can be any combination of any letters + any letters and any numbers (such as Figure 6 [A-Za-z][A-Za-z0-9__]*). The text pattern for the default text mark of a grammar symbol, as well as the grammar symbol itself.

[0095] Specifically, the server obtains text patterns of multiple preset text markers configured for the encoding language type, and for each text segment, pattern-matches the text patterns of the multiple preset text markers with the text segment to obtain matching text markers that match the text segment. The matching text markers that match each text segment are then aggregated to obtain a text marker sequence. In a specific application, by pattern-matching the text patterns of the multiple preset text markers with the text segment, a matching text pattern that matches the text segment can be determined, and then, based on the text markers indicated by the matching text patterns, a matching text marker that matches the text segment can be determined.

[0096] In this embodiment, by obtaining text patterns of multiple preset text markers configured for the coding language type, the text patterns of multiple preset text markers can be used to perform pattern matching on each text fragment respectively. Through text pattern matching, accurate determination of the matching text markers that match the text fragments is achieved, and then a text marker sequence can be obtained based on the matching text markers that match each text fragment.

[0097] In one embodiment, the text patterns of multiple preset text markers are pattern matched with the text segments respectively, and the matching text markers that match the text segments are obtained, including:

[0098] Performing pattern matching on the text patterns of the plurality of preset text markers and the text form of the text segment, respectively, and determining a matching text pattern that matches the text form from the text patterns of the plurality of preset text markers;

[0099] According to the text mark indicated by the matching text pattern, a matching text mark that matches the text segment is obtained.

[0100] Here, the text form refers to the form of a text fragment, which can be the text fragment itself or the grammatical form of the text fragment. For example, when the text fragment is a specific programming character or grammatical symbol, the text form can specifically be the text fragment itself. For another example, when the text fragment is an integer, a name, a string, etc., the text form refers to the grammatical form of the text fragment. For example, the grammatical form of an integer can be a combination of any numbers. The grammatical form of a name can be any combination of any letters + any letters and any numbers. The grammatical form of a string can be any combination of any letters.

[0101] Specifically, the server can determine a matching text pattern that matches the text form from the text patterns of multiple preset text markers by matching the text patterns of the multiple preset text markers with the text form of the text fragment, and then obtain a matching text marker that matches the text fragment based on the text marker indicated by the matching text pattern.

[0102] In a specific application, the matching text pattern that matches the text form can specifically be a text pattern that is identical to the text form. For example, in the case where the text form of a text fragment is the text fragment itself, the matching text pattern that matches the text form refers to a text pattern that is identical to the text form. It is understandable that the matching text pattern that matches the text form can specifically also be a text pattern that is similar to the text form. For example, in the case where the text form of a text fragment is the grammatical form of the text fragment, the matching text pattern that matches the text form refers to a text pattern that is similar to the text form. For example, when the text fragment is an integer, and the text form of the integer is actually a combination of any numbers, then the text pattern can also be represented as a combination of any numbers.

[0103] In a specific application, after determining a matching text pattern that matches the text form, the server needs to determine the number of text markers indicated by the matching text pattern. If the number of text markers indicated by the matching text pattern is a single one, the server can directly use the text marker indicated by the matching text pattern as the matching text marker that matches the text segment. If the number of text markers indicated by the matching text pattern is at least two, the server needs to compare the at least two text markers to determine the matching text marker that matches the text segment from the at least two text markers.

[0104] In this embodiment, by performing pattern matching on the text patterns of multiple preset text markers with the text form of the text fragment respectively, the matching text pattern that matches the text form can be determined by text pattern matching, and then the matching text marker that matches the text fragment can be determined based on the text marker indicated by the matching text pattern.

[0105] In one embodiment, obtaining a matching text marker that matches a text segment according to the text marker indicated by the matching text pattern includes:

[0106] When the number of text tokens indicated by the matching text pattern is at least two, determining a text token priority of each of the at least two text tokens;

[0107] Based on the text marker priorities of the at least two text markers, a matching text marker that matches the text segment is determined from the at least two text markers.

[0108] The text tag priority refers to the priority of a preset text tag during pattern matching and can be configured based on the actual application scenario. It is understood that different preset text tags have different priorities. For example, a preset text tag representing a specific programming character typically has a higher priority than a preset text tag representing a grammatical form.

[0109] Specifically, when the number of text markers indicated by the matching text pattern is at least two, the server will determine the text marker priority of each of the at least two text markers, sort the at least two text markers based on the text marker priority of each of the at least two text markers, and use the text marker with the highest priority as the matching text marker that matches the text fragment.

[0110] In specific applications, the priority of the preset text marker representing a specific programming character is usually higher than the preset text marker representing a grammatical form. Therefore, when the text identifiers indicated by the matching text pattern are respectively the preset text marker representing a specific programming character and the preset text marker representing a grammatical form, the server will use the preset text marker representing the specific programming character as the matching text marker that matches the text fragment.

[0111] In a specific application, such as Figure 7 As shown, the program type of this embodiment is described by taking the structured query language and the text fragment "SELECT" as an example. Figure 6 It can be seen from the multiple preset text tags configured for the structured query language and the text patterns of the multiple preset text tags that by performing pattern matching on the text patterns of the multiple preset text tags with the text form of the text fragment "SELECT", the matching text patterns that match the text form can be determined from the text patterns of the multiple preset text tags, including "SELECT", "[A-Za-z] [A-Za-z0-9_]* and "`(\\.|``|[^`\\])*`|\``(\\.|\``|[^``\\])*\``", and the number of text tags indicated by the matching text pattern is three, namely "SELECT", "NAME" and "STRING". The server will determine the text tag priority of each of the three text tags, and sort the three text tags based on the text tag priority of each of the three text tags, and use the text tag "SELECT" with the highest priority (that is, the preset text tag representing a specific programming character) as the matching text tag that matches the text fragment "SELECT".

[0112] In this embodiment, when the number of text markers indicated by the matching text pattern is at least two, by first determining the text marker priority of each of the at least two text markers, the text marker priority of each of the at least two text markers can be utilized to accurately determine the matching text marker that matches the text fragment from the at least two text markers through priority sorting.

[0113] In one embodiment, obtaining a text marker sequence according to the matching text markers of each text segment includes:

[0114] For each text segment, generating a text marker element representing the text segment according to the text segment and a matching text marker that matches the text segment;

[0115] The text marker elements representing each text fragment are arranged in the order of the position of each text fragment in the value of the key-value pair to obtain a text marker sequence.

[0116] The text mark element refers to an element used to represent a text segment in a text mark sequence.

[0117] Specifically, for each text fragment, the server will generate a text marker element representing the text fragment based on the text fragment and the matching text marker that matches the text fragment, and then arrange the text marker elements representing each text fragment in the order of the position of each text fragment in the value of the key-value pair to obtain a text marker sequence.

[0118] In a specific application, the server may combine the text segment and the matching text marker that matches the text segment, and use the combined segment as a text marker element representing the text segment. In a specific application, when combining the text segment and the matching text marker that matches the text segment, the combination may be a matching text marker + text segment, or a text segment + matching text marker. This embodiment does not limit the specific organization form. It is understood that the server may also directly use the matching text marker that matches the text segment as a text marker element representing the text segment.

[0119] In a specific application, such as Figure 8 As shown in the figure, this step is explained by taking the string "select 1 fromadmin where ms = 'test' or '1' = '1'" as an example. The string "select 1 from admin where ms = 'test' or '1' = '1'" can be split into 12 text segments, namely "select", "1", "from", "admin", "where", "ms", "=", "'test'", "or", "'1'", "=", "1'". Figure 6The multiple preset text tags configured for the structured query language and the text patterns of the multiple preset text tags are shown. After pattern matching with the text fragments respectively, the matching text tags that match each text fragment can be obtained. By combining the text fragments and the matching text tags that match the text fragments, the text tag elements that represent the text fragments can be obtained. The text identification elements that represent each text fragment are arranged in the order of the position of each text fragment in the value of the key-value pair to obtain a text tag sequence.

[0120] In this embodiment, for each text fragment, a text marker element representing the text fragment is generated by utilizing the text fragment and the matching text marker that matches the text fragment, thereby achieving accurate representation of the text fragment. Furthermore, the text marker elements representing each text fragment can be arranged according to the position order of each text fragment in the value of the key-value pair, thereby obtaining a text marker sequence that can accurately represent the value in the key-value pair.

[0121] In one embodiment, encoding the text tag sequence to obtain text tag features of the value in the key-value pair includes:

[0122] Encode each text mark element in the text mark sequence separately to obtain the encoding features of each text mark element;

[0123] Based on the attention mechanism, the encoding features of each text mark element are convolved to obtain the text mark features of the value in the key-value pair.

[0124] Among them, the attention mechanism is a resource allocation scheme that allocates computing resources to more important tasks when computing power is limited, while solving the problem of information overload. In neural network learning, generally speaking, the more parameters a model has, the stronger its expressive power is, and the larger the amount of information stored by the model is, but this will bring about the problem of information overload. Then, by introducing the attention mechanism, focusing on the information that is more critical to the current task among the numerous input information, reducing the attention to other information, and even filtering out irrelevant information, the problem of information overload can be solved, and the efficiency and accuracy of task processing can be improved. In this embodiment, the focus is on the more critical information in the respective encoding features of each text mark element to obtain text mark features that can accurately characterize the values in the key-value pairs.

[0125] Specifically, the server will encode each text marker element in the text marker sequence separately to obtain the encoding features of each text marker element, and then based on the attention mechanism, perform convolution processing on the encoding features of each text marker element to obtain the text marker features of the value in the key-value pair.

[0126] In a specific application, when performing convolution processing on the encoding features of each text marker element based on the attention mechanism, the server can combine the encoding features of each text marker element into an encoding feature sequence, input the encoding feature sequence into a pre-trained attention model, and use the attention model to perform semantic understanding on the encoding feature sequence to obtain the text marker features of the value in the key-value pair. In a specific application, the pre-trained attention model can be trained according to the actual application scenario, and this is not specifically limited in this embodiment.

[0127] In specific applications, the attention mechanism can be a multi-head attention mechanism. For each attention mechanism in the multi-head attention mechanism, the terminal will perform convolution processing on the respective encoding features of each text mark element based on the targeted attention mechanism to obtain the text mark features corresponding to the targeted attention mechanism. By fusing the text mark features corresponding to each attention mechanism, the text mark features of the value in the key-value pair can be obtained.

[0128] In this embodiment, by encoding each text marker element in the text marker sequence separately, the encoding features obtained by encoding can be used to accurately represent the text marker elements. Furthermore, by performing convolution processing on the encoding features of each text marker element based on the attention mechanism, the semantics of each text marker element can be fully integrated to obtain the text marker features of the values in the key-value pairs, thereby achieving accurate representation of the values in the key-value pairs.

[0129] In one embodiment, based on the attention mechanism, convolution processing is performed on the encoding features of each text mark element to obtain the text mark features of the value in the key-value pair, including:

[0130] Based on the transformation matrix of the attention mechanism, the encoding features of each text mark element are linearly transformed to obtain the query vector, key vector and value vector corresponding to each text mark element;

[0131] Calculate the attention weight according to the query vector and value vector corresponding to each text mark element to obtain the attention weight corresponding to each text mark element;

[0132] Based on the attention weight corresponding to each text mark element, the value vector corresponding to each text mark element is weighted to obtain the text mark feature of the value in the key-value pair.

[0133] Among them, the transformation matrix of the attention mechanism includes a query vector transformation matrix, a key vector transformation matrix and a value vector transformation matrix, which correspond to the query vector, key vector and value vector respectively. The implementation of the attention mechanism mainly relies on three vectors: query vector, key vector and value vector. These three vectors are linear transformations of the input data. In this embodiment, the input data includes the respective encoding features of each text mark element. Among them, the functions and interaction methods of the query vector, key vector and value vector are as follows: Query vector: used to represent the current input element and used to query other related elements. Key vector: used to represent all input elements and used to be queried. Value vector: used to represent all input elements. When an element is queried, its corresponding value vector will be used to calculate the output.

[0134] Specifically, the server will perform linear transformation on the encoding features of each text marker element based on the query vector transformation matrix in the transformation matrix of the attention mechanism to obtain the query vector corresponding to each text marker element, and perform linear transformation on the encoding features of each text marker element based on the key vector transformation matrix in the transformation matrix of the attention mechanism to obtain the key vector corresponding to each text marker element, and perform linear transformation on the encoding features of each text marker element based on the value vector transformation matrix in the transformation matrix of the attention mechanism to obtain the value vector corresponding to each text marker element.

[0135] Specifically, based on the query vector, key vector, and value vector corresponding to each text marker element, the server can obtain the text marker feature of the value in the key-value pair through the interaction of the query vector, key vector, and value vector corresponding to each text marker element. Specifically, the server will perform attention weight calculation based on the query vector and value vector corresponding to each text marker element, obtain the attention weight corresponding to each text marker element, use the attention weight corresponding to each text marker element as the weighting coefficient of the value vector corresponding to each text marker element, and weight the value vector corresponding to each text marker element to obtain the text marker feature of the value in the key-value pair.

[0136] It can be understood that in the text mark feature, the weighted value vector corresponding to each text mark element is arranged in order according to the position of each text mark element in the text mark sequence, that is, the text mark feature is a combination of the weighted value vectors corresponding to each text mark element.

[0137] In specific applications, the server can calculate the attention weight by scaling the dot product, that is, first calculate the similarity between the query vector and the value vector corresponding to each text marker element, obtain the dot product, and then scale the dot product to obtain the attention weight corresponding to each text marker element.

[0138] In this embodiment, based on the transformation matrix of the attention mechanism, the query vector, key vector and value vector corresponding to each text marker element can be determined by linear transformation. By calculating the attention weight according to the query vector and value vector corresponding to each text marker element, the attention weight corresponding to each text marker element can be obtained. Therefore, based on the attention weight corresponding to each text marker element, the value vector corresponding to each text marker element can be weighted, so that the semantics of each text marker element can be fully integrated, and the text marker features of the values in the key-value pairs can be obtained, thereby achieving accurate representation of the values in the key-value pairs.

[0139] In one embodiment, identifying the text category to which the value in the key-value pair belongs based on the text marker feature includes:

[0140] Get the language structure characteristics of the value in the key-value pair;

[0141] Identify the text category to which the value in the key-value pair belongs based on text mark features and language structure features.

[0142] Language structure features refer to features of the language structure used to describe the value in a key-value pair. For example, a language structure feature can specifically be a feature of the encoding language type used to describe the value in a key-value pair, known as a language category feature. Another example is a language structure feature that specifically describes the statement category used to describe the value in a key-value pair, known as a statement category feature. A statement category can specifically be the type of operation indicated by the value in the key-value pair, such as add, delete, or query. Another example is a language structure feature that specifically describes whether the value in a key-value pair includes a specific function, known as a function category feature. Specific functions can be configured based on actual application scenarios. For example, for structured query language, specific functions can specifically be CALL functions, UNION functions, and so on. Another example is a language structure feature that specifically describes any combination of at least two of a language category feature, a statement category feature, and multiple function category features.

[0143] Specifically, the server extracts features from the key-value pair based on the combination of linguistic structural features to obtain the linguistic structural features of the value. It then predicts the text category based on the text marker features and linguistic structural features, identifying the text category to which the value in the key-value pair belongs. In specific applications, the combination of linguistic structural features determines the type of features the server needs to extract from the key-value pair.

[0144] In a specific application, when the language structure feature is only one of the language category feature, the statement category feature, and the multiple function category features, the server only needs to determine the information corresponding to the feature based on the value in the key-value pair. For example, when the language structure feature is only the language category feature, the server only needs to determine the coding language type. For another example, when the language structure feature is only the statement category feature, the server only needs to identify the statement category. For another example, when the language structure feature is only a certain function category, the server only needs to identify whether there is a text fragment representing the function category in the value in the key-value pair. When the language structure feature is any combination of at least two of the language category feature, the statement category feature, and the multiple function category features, the server needs to first determine the information corresponding to each feature separately, and then obtain the language structure feature based on the information corresponding to each feature.

[0145] In this embodiment, by obtaining the language structure features of the values in the key-value pair, the text category to which the values in the key-value pair belong is identified based on the text mark features and the language structure features. The text mark features and the language structure features can be combined at the same time to identify the text category to which the values in the key-value pair belong, thereby improving the recognition accuracy.

[0146] In one embodiment, the language structure features for obtaining the value in the key-value pair include:

[0147] Identify the statement category of the value in the key-value pair and determine the function category information of the value in the key-value pair;

[0148] The coding language type, statement category, and function category information are respectively mapped to preset language structure codes to obtain the language structure characteristics of the value in the key-value pair.

[0149] Function category information refers to the function category information included in the value of the key-value pair. For example, the function category information can be in the form of "Include A Function" or "Exclude A Function". The specific function category to be identified can be configured according to the actual application scenario. For example, the specific function category can be "CALL Function" or "UNION Function".

[0150] Specifically, based on determining the coding language type of the value in the key-value pair, the server can obtain the statement identifier of each statement under the coding language type, match the statement identifier of each statement with the value in the key-value pair to identify the statement category of the value in the key-value pair, and obtain the function identifier of each required identification function category under the coding language type, match the function identifier of each required identification function category with the value in the key-value pair to determine the function category information of the value in the key-value pair.

[0151] A statement identifier is a string that uniquely identifies the statement type. For example, in Structured Query Language (SQL), the statement identifier for a query statement is "SELECT," the statement identifiers for an add statement might be "INSERT" or "ADD," and the statement identifier for a delete statement might be "DELETE." A function identifier is a string that uniquely identifies the function type. For example, the function identifier for a CALL function might be "CALL," and the function identifier for a UNION function might be "UNION."

[0152] Specifically, based on the information of the coding language type, statement type and function category, the server will map the coding language type, statement type and function category information to preset language structure codes respectively to obtain the language structure characteristics of the value in the key-value pair. In specific applications, the preset language structure codes mapped to different coding language types are different, but the code size is the same. The preset language structure codes mapped to different statement types are different, but the code size is the same. For different function category information, the mapped preset language structure codes are different, but the code size is the same. Among them, the preset language structure codes mapped to each coding language type, each statement type and different function category information can be configured according to the actual application scenario, and are not specifically limited here in this embodiment.

[0153] In this embodiment, by identifying the statement category of the value in the key-value pair and determining the function category information of the value in the key-value pair, the coding language type, statement category and function category information are respectively mapped to preset language structure coding. The language structure characteristics of the value in the key-value pair can be determined by first extracting the feature information and then performing coding mapping. The obtained language structure characteristics, because they combine the coding language type, statement category and function category information, can accurately represent the language structure of the value in the key-value pair.

[0154] In one embodiment, identifying the text category to which the value in the key-value pair belongs based on the text mark feature and the language structure feature includes:

[0155] Predict text categories based on text mark features and language structure features, and obtain the probability that the value in the key-value pair belongs to each preset text category;

[0156] The text category to which the value in the key-value pair belongs is determined based on the probability that the value in the key-value pair belongs to each preset text category.

[0157] Specifically, the server will predict the text category based on the text marker features and language structure features, obtain the probability that the value in the key-value pair belongs to each preset text category, and determine the text category to which the value in the key-value pair belongs based on the probability that the value in the key-value pair belongs to each preset text category. In a specific application, the server can input the text marker features and language structure features into a pre-trained text classification model, identify the text category to which the value in the key-value pair belongs through the pre-trained text classification model, and use the text category output by the text classification model as the text category to which the value in the key-value pair belongs. Among them, the pre-trained text classification model can be trained according to the actual application scenario. In this embodiment, no specific limitation is made here.

[0158] In a specific application, such as Figure 9 As shown in FIG, a flow chart is used to illustrate the text category to which the value in the key-value pair involved in this embodiment belongs. Figure 9 As shown, the server will input the text mark features and language structure features into the pre-trained text classification model for model feature fusion (that is, through the pre-trained text classification model, the text mark features and language structure features are fused), and obtain the probability that the value in the key-value pair belongs to each preset text category. According to the probability that the value in the key-value pair belongs to each preset text category, the text category to which the value in the key-value pair belongs is determined.

[0159] Among them, such as Figure 9 As shown in the figure, language structure features can be obtained by inputting feature information such as feature 1: language category (specifically, one of SQL, JS, and Shell), feature 2: statement category (specifically, one of SELECT, ADD, etc.), function category information (specifically, feature 3: CALL feature and feature 4: UNION feature) into the feature model. Figure 9 Other feature information not shown in the figure is represented by feature N. It is understandable that the feature model can map these feature information into different preset language structure feature codes.

[0160] Among them, such as Figure 9As shown, the text mark feature can be obtained by segmenting the message data into text, digitally encoding, and processing it using an attention model. The specific method of segmenting the text is to parse the message data into multiple key-value pairs in text form, and for each key-value pair, determine the encoding language type that matches the key-value pair, and segment the value in the key-value pair into multiple text fragments. Based on the encoding language type, each text fragment is mapped to a preset text mark to obtain a text mark sequence. The specific method of digital encoding is to encode the text mark sequence to obtain a digital encoding sequence of the value in the key-value pair. The specific method of processing using an attention model is to input the digital encoding sequence into the attention model to obtain the text mark feature of the value in the key-value pair.

[0161] In this embodiment, text category prediction is performed based on text mark features and language structure features, and the probability that the value in the key-value pair belongs to each preset text category can be obtained. Then, based on the probability that the value in the key-value pair belongs to each preset text category, the text category to which the value in the key-value pair belongs can be determined.

[0162] In one embodiment, determining the text category to which the value in the key-value pair belongs based on the probability that the value in the key-value pair belongs to each preset text category includes:

[0163] Determine the candidate text category with the highest probability from the probabilities that the values in the key-value pairs belong to each preset text category;

[0164] When the probability that the value in the key-value pair belongs to the candidate text category is greater than a preset probability threshold, the candidate text category is used as the text category to which the value in the key-value pair belongs.

[0165] Specifically, the server may determine the candidate text category with the highest probability from the probabilities that the value in the key-value pair belongs to each preset text category. When the probability that the value in the key-value pair belongs to the candidate text category is greater than a preset probability threshold, the candidate text category is used as the text category to which the value in the key-value pair belongs. The preset probability threshold can be configured according to the actual application scenario. For example, the preset probability threshold can be specifically 80%.

[0166] In a specific application, when the probability that the value in the key-value pair belongs to the candidate text category is less than or equal to a preset probability threshold, the server will further compare the probability that the value in the key-value pair belongs to the candidate text category with the second highest probability among the probabilities that the value in the key-value pair belongs to each preset text category, and determine the difference between the two. When the probability difference is greater than or equal to the difference threshold, the candidate text category can also be used as the text category to which the value in the key-value pair belongs. The difference threshold can be configured according to the actual application scenario. For example, the difference threshold can be specifically 20%.

[0167] In a specific application, the steps of this embodiment are described by taking each preset text category including normal text and abnormal text as an example. Figure 10 As shown, assuming that the probability that the value in the key-value pair belongs to normal text is 40%, and the probability that the value in the key-value pair belongs to abnormal text is 60%, the server can determine that the candidate text category with the largest probability (60%) is abnormal text. When the probability that the value in the key-value pair belongs to abnormal text (60%) is less than the preset probability threshold (80%), the server will further compare the probability that the value in the key-value pair belongs to abnormal text (60%) and the probability that the value in the key-value pair belongs to normal text (40%). When the difference between the two, that is, the probability difference (20%) is greater than or equal to the difference threshold (20%), it is determined that the text category to which the value in the key-value pair belongs is abnormal text.

[0168] In this embodiment, based on the probability that the value in the key-value pair belongs to each preset text category, by determining the candidate text category with the highest probability, and comparing the probability that the value in the key-value pair belongs to the candidate text category with the preset probability threshold, it is possible to accurately determine the text category to which the value in the key-value pair belongs.

[0169] In one embodiment, the application of the message data processing method of this application to network security protection is used as an example for illustration. The inventor believes that traditional network security protection mainly relies on regular expression rules for detection. Generally speaking, the processing of rules requires a lot of manual intervention. If the expression of more complex rules is more complex, more complex regular expressions are required, which is prone to errors, resulting in false detection and high maintenance costs.

[0170] Based on this, the present application proposes a message data processing method that integrates an attention model and a feature model to determine the category of the message data. This is different from the general field of using attention models, which directly performs word segmentation on the text to be detected, and then encodes and inputs it into the model for training. In the present application, the message data is first deeply parsed and decoded to obtain multiple key-value pairs in the decoded text form. For each key-value pair, the value in the key-value pair is grammatically segmented to obtain a text mark sequence representing the value in the key-value pair. The text mark sequence is then encoded to obtain a digital code. The attention model is then used for semantic understanding to obtain the text mark features of the value in the key-value pair. At the same time, according to the feature engineering method, based on the value in the key-value pair, the language structure features are obtained. Finally, the text mark features and the language structure features are fused and judged using a feature model. The text mark features and the language structure features are combined to determine the text category to which the value in the key-value pair belongs. Then, the category of the message data is determined according to the text category to which the values of the multiple key-value pairs belong.

[0171] It can be understood that the message data processing method in the present application, based on parsing the message data and obtaining multiple key-value pairs in text form, can determine the text category to which the value in the key-value pair belongs through steps such as segmentation, mapping, encoding, and identification of text categories, and then use the text category to which the values of each of the multiple key-value pairs belong to determine the category of the message data, that is, the category of the message data is determined by analyzing the text content of the message data without the need to configure complex regular expression rules, which can improve the accuracy of message data category detection.

[0172] In one embodiment, Figure 11 As shown, taking the application of HTTP message as an example, the message data processing method of this application is explained through a flow chart, as shown in FIG. Figure 11 As shown, the message data processing method mainly includes the following steps:

[0173] 1. The server is used to receive network data packets, mainly used to receive HTTP messages sent from the client.

[0174] 2. After receiving the HTTP message, the server performs fine-grained parsing of the HTTP protocol, including URL parameter parsing, Head header parsing, Cookie parsing, Json parsing, and XML parsing. Through these parsing, a list of key-value pairs (key-value pairs) can be obtained, that is, multiple key-value pairs in text form.

[0175] 3. The parsed Value string (the value in the key-value pair) is then evaluated using a language model to determine the specific language type of the statement, such as SQL, JavaScript, or Shell. Strings that do not have a language model are removed.

[0176] Specifically, for each key-value pair, the server will determine the encoding language type that the key-value pair matches. It is understandable that for key-value pairs whose encoding language type cannot be determined, the server will choose not to process them, that is, remove the strings that cannot obtain the language model.

[0177] 4. Input the Value string (the value in the key-value pair) as text into the text segmentation module, perform text segmentation processing, and obtain a Token list (text token sequence).

[0178] Specifically, the server will divide the value in the key-value pair into multiple text segments, and based on the coding language type, map each text segment to a preset text marker to obtain a text marker sequence. In a specific application, the server will obtain the text patterns of multiple preset text markers configured for the coding language type, and for each text segment, perform pattern matching on the text patterns of multiple preset text markers with the text form of the text segment, and determine the matching text pattern that matches the text form from the text patterns of multiple preset text markers, and obtain the matching text marker that matches the text segment according to the text marker indicated by the matching text pattern. For each text segment, the server will generate a text marker element that represents the text segment based on the text segment and the matching text marker that matches the text segment, and arrange the text marker elements that represent each text segment in the order of the position of each text segment in the value of the key-value pair to obtain a text marker sequence.

[0179] In a specific application, when the number of text markers indicated by the matching text pattern is at least two, a text marker priority of each of the at least two text markers is determined, and based on the text marker priorities of each of the at least two text markers, a matching text marker that matches the text segment is determined from the at least two text markers. When the number of text markers indicated by the matching text pattern is only one, the text marker indicated by the matching text pattern is directly used as the matching text marker that matches the text segment.

[0180] 5. Digitally encode the obtained token list (text token sequence) to obtain a digital list, and input it into the attention model to obtain text token features.

[0181] Specifically, the server digitally encodes each text tag element in the text tag sequence, obtains the encoded features of each text tag element, aggregates the encoded features of each text tag element (actually digital codes), obtains a numerical list, and inputs the numerical list into the attention model to obtain the text tag features. It can be understood that inputting the numerical list into the attention model actually performs convolution processing on the encoded features of each text tag element based on the attention mechanism.

[0182] In a specific application, the process of the attention model for processing a list of numbers can be as follows: based on the transformation matrix of the attention mechanism, linearly transform the encoding features of each text marker element respectively to obtain the query vector, key vector and value vector corresponding to each text marker element, calculate the attention weight according to the query vector and value vector corresponding to each text marker element, obtain the attention weight corresponding to each text marker element, and based on the attention weight corresponding to each text marker element, weight the value vector corresponding to each text marker element to obtain the text marker feature of the value in the key-value pair.

[0183] 6. Extract language structure features through feature engineering, mainly including language category features, statement category features, CALL features, UNION features, etc.

[0184] Specifically, the server will identify the statement category of the value in the key-value pair, and determine the function category information of the value in the key-value pair, and map the coding language type, statement category and function category information to preset language structure coding to obtain the language structure characteristics of the value in the key-value pair.

[0185] 7. Input the text landmark features obtained by the attention model and the language structure features obtained by feature engineering into the fusion model (i.e., the pre-trained text classification model) to obtain the final detection result, i.e., the text category to which the value of the key-value pair belongs, and feedback the detection result.

[0186] Specifically, the server inputs the text marker features and language structure features into the fusion model (i.e., the pre-trained text classification model) to obtain the final detection result, namely, the text category to which the value in the key-value pair belongs. In specific applications, the processing flow for the fusion model to perform text classification prediction is as follows: text category prediction is performed based on the text marker features and language structure features, and the probability of the value in the key-value pair belonging to each preset text category is obtained. From the probabilities of the value in the key-value pair belonging to each preset text category, the candidate text category with the highest probability is determined. When the probability that the value in the key-value pair belongs to the candidate text category is greater than the preset probability threshold, the candidate text category is used as the text category to which the value in the key-value pair belongs.

[0187] 8. After receiving the detection result, the server blocks or releases the HTTP message based on the result.

[0188] Specifically, after obtaining the text category of each key-value pair's value, the server determines the HTTP message's category based on the text categories of the key-value pairs' values, thereby blocking or allowing the HTTP message. It is understood that when the HTTP message is classified as a normal message, the server will allow the HTTP message to pass. When the HTTP message is classified as an abnormal message, the server will block the HTTP message.

[0189] It should be understood that, although the various steps in the flowcharts involved in the various embodiments described above are displayed in sequence according to the instructions of the arrows, these steps are not necessarily executed in sequence in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and these steps can be executed in other orders. Moreover, at least a portion of the steps in the flowcharts involved in the various embodiments described above can include multiple steps or multiple stages, and these steps or stages are not necessarily executed and completed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a portion of steps or stages in other steps.

[0190] Based on the same inventive concept, embodiments of the present application further provide a message data processing device for implementing the message data processing method involved above. The implementation solution provided by this device is similar to the implementation solution described in the above method. Therefore, the specific limitations of one or more message data processing device embodiments provided below can be found in the above-mentioned limitations of the message data processing method and will not be repeated here.

[0191] In one embodiment, Figure 12 As shown, a message data processing device is provided, comprising: a message data acquisition module 1202, a text mark generation module 1204, an encoding module 1206, a category identification module 1208 and a processing module 1210, wherein:

[0192] The message data acquisition module 1202 is used to acquire message data and parse the message data into multiple key-value pairs in text form;

[0193] The text tag generation module 1204 is configured to determine, for each key-value pair, the type of coding language that the key-value pair matches, and to segment the value in the key-value pair into multiple text segments. Based on the coding language type, each text segment is mapped to a preset text tag to obtain a text tag sequence.

[0194] An encoding module 1206 is used to encode the text tag sequence to obtain the text tag features of the values in the key-value pairs;

[0195] Category identification module 1208, for identifying the text category to which the value in the key-value pair belongs based on the text mark feature;

[0196] The processing module 1210 is configured to determine the category of the message data according to the text categories to which the values of the multiple key-value pairs belong.

[0197] The above-mentioned message data processing device, based on the acquisition of message data, can obtain multiple key-value pairs in the form of parsed text by deeply parsing the message data. For each key-value pair, after determining the coding language type that matches the key-value pair and dividing the value in the key-value pair into multiple text fragments, each text fragment is mapped to a preset text flag with reference to the coding language type, so as to obtain a text flag sequence that can represent the value in the key-value pair. By encoding the text flag sequence, the features in the text flag sequence can be extracted to obtain the text flag features of the value in the key-value pair. The text flag features can then be used to accurately identify the text category to which the value in the key-value pair belongs. On this basis, according to the text category to which the values of each of the multiple key-value pairs belong, the category of the message data can be accurately determined. The entire process, based on parsing the message data and obtaining multiple key-value pairs in text form, can determine the text category to which the value in the key-value pair belongs through steps such as segmentation, mapping, encoding, and identification of text categories, and then use the text categories to which the values of multiple key-value pairs belong to determine the category of the message data. That is, the category of the message data is determined by analyzing the text content of the message data without the need to configure complex regular expression rules, which can improve the accuracy of message data category detection.

[0198] In one embodiment, the text marker generation module is also used to obtain text patterns of multiple preset text markers configured for the coding language type, and for each text fragment, the text patterns of the multiple preset text markers are pattern matched with the text fragment to obtain matching text markers that match the text fragment, and a text marker sequence is obtained based on the matching text markers that match each text fragment.

[0199] In one embodiment, the text marker generation module is also used to perform pattern matching on the text patterns of multiple preset text markers with the text form of the text fragment, determine the matching text pattern that matches the text form from the text patterns of the multiple preset text markers, and obtain the matching text marker that matches the text fragment based on the text marker indicated by the matching text pattern.

[0200] In one embodiment, the text marker generation module is also used to determine the text marker priority of each of the at least two text markers when the number of text markers indicated by the matching text pattern is at least two, and based on the text marker priority of each of the at least two text markers, determine a matching text marker that matches the text fragment from the at least two text markers.

[0201] In one embodiment, the text marker generation module is also used to generate a text marker element representing the text fragment for each text fragment based on the text fragment and the matching text marker that matches the text fragment, and arrange the text marker elements representing each text fragment in the order of the position of each text fragment in the value of the key-value pair to obtain a text marker sequence.

[0202] In one embodiment, the text marker generation module is also used to encode each text marker element in the text marker sequence separately to obtain the encoding features of each text marker element. Based on the attention mechanism, the encoding features of each text marker element are convolved to obtain the text marker features of the value in the key-value pair.

[0203] In one embodiment, the text marker generation module is also used to perform linear transformation on the encoding features of each text marker element based on the transformation matrix of the attention mechanism, and obtain the query vector, key vector and value vector corresponding to each text marker element. The attention weight is calculated according to the query vector and value vector corresponding to each text marker element to obtain the attention weight corresponding to each text marker element. Based on the attention weight corresponding to each text marker element, the value vector corresponding to each text marker element is weighted to obtain the text marker feature of the value in the key-value pair.

[0204] In one embodiment, the category identification module is further configured to obtain language structure features of the value in the key-value pair, and identify the text category to which the value in the key-value pair belongs based on the text mark features and the language structure features.

[0205] In one embodiment, the category identification module is also used to identify the statement category of the value in the key-value pair, and determine the function category information of the value in the key-value pair, and map the coding language type, statement category and function category information to preset language structure coding to obtain the language structure characteristics of the value in the key-value pair.

[0206] In one embodiment, the category identification module is also used to predict text categories based on text mark features and language structure features, obtain the probability that the value in the key-value pair belongs to each preset text category, and determine the text category to which the value in the key-value pair belongs based on the probability that the value in the key-value pair belongs to each preset text category.

[0207] In one embodiment, the category identification module is also used to determine the candidate text category with the highest probability from the probabilities that the values in the key-value pair belong to each preset text category. When the probability that the value in the key-value pair belongs to the candidate text category is greater than a preset probability threshold, the candidate text category is used as the text category to which the value in the key-value pair belongs.

[0208] Each module in the above-mentioned message data processing device can be implemented in whole or in part through software, hardware, or a combination thereof. Each module can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in the computer device in the form of software, so that the processor can call and execute the corresponding operations of each module.

[0209] In one embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as follows: Figure 13 As shown. The computer device includes a processor, a memory, an input / output interface (Input / Output, abbreviated as I / O) and a communication interface. The processor, memory and input / output interface are connected via a system bus, and the communication interface is connected to the system bus via the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and computer program in the non-volatile storage medium. The database of the computer device is used to store data such as text modes and preset coding tables. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal via a network connection. When the computer program is executed by the processor, a message data processing method is implemented.

[0210] Those skilled in the art will understand that Figure 13 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than shown in the figure, or combine certain components, or have a different component arrangement.

[0211] In one embodiment, a computer device is further provided, including a memory and a processor. The memory stores a computer program, and the processor implements the steps in the above method embodiments when executing the computer program.

[0212] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments are implemented.

[0213] In one embodiment, a computer program product is provided, including a computer program, which implements the steps in the above method embodiments when executed by a processor.

[0214] It should be noted that the data involved in this application (including but not limited to data used for analysis, stored data, etc.) are all authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant regulations.

[0215] Those skilled in the art will appreciate that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the above-mentioned embodiments. In particular, any reference to memory, database, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The databases involved in the various embodiments provided herein may include at least one of a relational database and a non-relational database. Non-relational databases may include, but are not limited to, distributed databases based on blockchains. The processors involved in the various embodiments provided herein may be, but are not limited to, general-purpose processors, central processing units (CPUs), graphics processing units (GPUs), digital signal processors (DSPs), programmable logic devices (PLDs), data processing logic devices based on quantum computing, and the like.

[0216] The technical features of the above embodiments can be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0217] The above-described embodiments merely represent several implementation methods of the present application. While the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the present application. It should be noted that a person of ordinary skill in the art may make various modifications and improvements without departing from the spirit of the present application, and these modifications and improvements fall within the scope of protection of the present application. Therefore, the scope of protection of the present application shall be determined by the appended claims.

Claims

1. A message data processing method, characterized in that: The method comprises: Obtaining message data, and parsing the message data into multiple key-value pairs in text form; For each key-value pair, determining a coding language type that matches the key-value pair, dividing the value in the key-value pair into a plurality of text segments, and mapping each of the text segments to a preset text tag based on the coding language type to obtain a text tag sequence; Encoding the text mark sequence to obtain text mark features of the values in the key-value pairs; Identifying the text category to which the value in the key-value pair belongs based on the text mark feature; The category of the message data is determined according to the text category to which the values of each of the multiple key-value pairs belong.

2. The method according to claim 1, characterized in that Mapping each of the text segments to a preset text marker based on the coding language type to obtain a text marker sequence includes: Obtaining a text mode of a plurality of preset text symbols configured for the coding language type; For each of the text segments, pattern matching is performed on the text segments with the text patterns of the plurality of preset text markers to obtain matching text markers that match the text segments; A text marker sequence is obtained according to the matching text markers of each text segment.

3. The method according to claim 2, characterized in that The text patterns of the plurality of preset text marks are matched with the text fragments to obtain matching text marks that match the text fragments, including: Performing pattern matching on the text patterns of the plurality of preset text markers and the text form of the text segment, respectively, and determining a matching text pattern that matches the text form from the text patterns of the plurality of preset text markers; According to the text mark indicated by the matching text pattern, a matching text mark that matches the text segment is obtained.

4. The method according to claim 3, characterized in that Obtaining a matching text marker that matches the text segment according to the text marker indicated by the matching text pattern includes: When the number of text markers indicated by the matching text pattern is at least two, determining a text marker priority of each of the at least two text markers; Based on the text marker priorities of the at least two text markers, a matching text marker that matches the text segment is determined from the at least two text markers.

5. The method according to claim 2, characterized in that Obtaining a text marker sequence according to the matching text markers of each text segment includes: For each of the text segments, generating a text marker element representing the text segment according to the text segment and a matching text marker that matches the text segment; The text marker elements representing each of the text segments are arranged in the order of the positions of the values of each of the text segments in the key-value pairs to obtain a text marker sequence.

6. The method according to claim 1, characterized in that The encoding of the text mark sequence to obtain the text mark feature of the value in the key-value pair includes: Encoding each text mark element in the text mark sequence to obtain a respective encoding feature of each text mark element; Based on the attention mechanism, convolution processing is performed on the encoding features of each text mark element to obtain the text mark features of the value in the key-value pair.

7. The method according to claim 6, characterized in that The attention mechanism is based on performing convolution processing on the encoding features of each of the text mark elements to obtain the text mark features of the value in the key-value pair, including: Based on the transformation matrix of the attention mechanism, linearly transform the encoding features of each of the text mark elements to obtain the query vector, key vector and value vector corresponding to each of the text mark elements; Calculate the attention weight according to the query vector and the value vector corresponding to each of the text mark elements to obtain the attention weight corresponding to each of the text mark elements; Based on the attention weight corresponding to each of the text mark elements, the value vector corresponding to each of the text mark elements is weighted to obtain the text mark feature of the value in the key-value pair.

8. The method according to any one of claims 1 to 7, characterized in that The identifying, based on the text mark feature, the text category to which the value in the key-value pair belongs includes: Obtaining language structure characteristics of the value in the key-value pair; The text category to which the value in the key-value pair belongs is identified according to the text mark feature and the language structure feature.

9. The method according to claim 8, characterized in that The language structure feature of obtaining the value in the key-value pair includes: Identifying a statement category of a value in the key-value pair, and determining function category information of the value in the key-value pair; The coding language type, the statement category, and the function category information are respectively mapped to preset language structure codes to obtain language structure features of the values in the key-value pairs.

10. The method according to claim 8, characterized in that The identifying, based on the text mark feature and the language structure feature, the text category to which the value in the key-value pair belongs comprises: Predicting text categories based on text marker features and the language structure features to obtain probabilities that the values in the key-value pairs belong to each preset text category; The text category to which the value in the key-value pair belongs is determined according to the probability that the value in the key-value pair belongs to each preset text category.

11. The method according to claim 10, characterized in that Determining the text category to which the value in the key-value pair belongs according to the probability that the value in the key-value pair belongs to each preset text category includes: Determining the candidate text category with the greatest probability from the probabilities that the values in the key-value pairs belong to the preset text categories; When the probability that the value in the key-value pair belongs to the candidate text category is greater than a preset probability threshold, the candidate text category is used as the text category to which the value in the key-value pair belongs.

12. A message data processing device, characterized in that: The device comprises: A message data acquisition module, used to acquire message data and parse the message data into multiple key-value pairs in text form; a text tag generation module, configured to determine, for each key-value pair, a coding language type that matches the key-value pair, and segment the value in the key-value pair into a plurality of text segments; and based on the coding language type, map each of the text segments to a preset text tag to obtain a text tag sequence; An encoding module, configured to encode the text mark sequence to obtain text mark features of the values in the key-value pairs; A category identification module, configured to identify the text category to which the value in the key-value pair belongs based on the text mark feature; The processing module is used to determine the category of the message data according to the text category to which the values of each of the multiple key-value pairs belong.

13. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 11 are implemented.

14. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 11 are implemented.

15. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 11 are implemented.

Images