Network traffic tracing method and device based on multi-modal data identification
By extracting traffic summary information and inferring structured data blocks of multimodal network traffic data, the problems of large amount of network traffic traceability and high resource utilization in the prior art are solved, and efficient multimodal data traffic traceability is achieved.
Patent Information
- Application Number
- CN202411863230.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-12-17
- Publication Date
- 2025-05-06
Smart Images
Figure CN119940526A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of information security technology, and in particular to a network traffic tracing method and device based on multimodal data recognition. Background Art
[0002] At present, network traffic tracing is an important part of post-event response to security incidents. Analyzing network traffic to achieve fine management and control of network traffic will help to fix vulnerabilities and risks and avoid secondary incidents, which is of great significance to improving information security.
[0003] In related technologies, deep packet inspection methods are used to collect traffic and packets at key network nodes, extract key information, and conduct in-depth analysis of the application layer content of data packets based on protocol analysis to identify specific network traffic or behavior. However, due to the high diversity and complexity of multimodal data, and the large amount of computing and storage resources required for the processing and analysis of multimodal data, the multimodal data analysis efficiency and query efficiency in complex scenarios are low. Therefore, how to achieve the analysis and traceability of multimodal data traffic under limited hardware resources is an urgent problem to be solved. Summary of the invention
[0004] The present invention provides a network traffic tracing method and device based on multimodal data identification, which are used to solve the defects of the prior art that the deep packet inspection method is used to trace the network traffic, the amount of calculation is large and the hardware resources are occupied, resulting in low efficiency in parsing and tracing the multimodal data traffic, thereby improving the tracing efficiency of the multimodal data traffic in complex scenarios.
[0005] The present invention provides a network traffic tracing method based on multimodal data identification, comprising: Extracting traffic summary information from multimodal network traffic data; wherein the multimodal network traffic data includes multiple data contents and multiple traceable information; Inferring the multiple data contents to obtain inference structured data, and generating an inference structured data block with the traffic summary information as the first data header and the inference structured data as the first data body; generating a traceability information block with the traffic summary information as the second data header and the traceability information as the second data body; An association is constructed according to the mapping relationship between the tracing information block and the inference structured data block to query the tracing information of the target network traffic.
[0006] According to a network traffic tracing method based on multimodal data identification provided by the present invention, the extraction of traffic summary information from multimodal network traffic data includes: Parsing the multimodal network traffic data and extracting quintuple information from the parsed traffic data; The traffic summary information is generated according to the quintuple information.
[0007] According to a network traffic tracing method based on multimodal data identification provided by the present invention, the multiple data contents include pictures, long videos and long audios; The reasoning on the multiple data contents to obtain the inferred structured data includes: The picture, the long video and the long audio are respectively segmented into fine-grained segments to obtain at least two of the picture, the fine-grained audio and the fine-grained video. The picture, fine-grained audio, and fine-grained video content are processed based on the inference big model to obtain the inference structured data.
[0008] According to a network traffic tracing method based on multimodal data identification provided by the present invention, the multimodal network traffic data is obtained through the following steps: Obtain network traffic data required for deep message detection based on the data traffic interface; The network traffic data is filtered and reorganized, multiple types of traffic features are extracted from the reorganized network traffic data, and the multimodal network traffic data is generated according to the multiple types of traffic features.
[0009] According to a network traffic tracing method based on multimodal data identification provided by the present invention, after constructing an association according to a mapping relationship between the tracing information block and the inference structured data block, the method further includes: Obtain keywords of target network traffic through user input; The keyword is traced based on the association to obtain target traceability information.
[0010] The present invention also provides a network traffic source tracing device based on multimodal data identification, comprising: An information extraction module, used to extract traffic summary information from multimodal network traffic data; wherein the multimodal network traffic data includes multiple data contents and multiple traceable information; A data block generation module is used to infer the multiple data contents to obtain inference structured data, and to generate an inference structured data block with the traffic summary information as the first data header and the inference structured data as the first data body; and to generate a traceability information block with the traffic summary information as the second data header and the traceability information as the second data body; The association module is used to construct an association body according to the mapping relationship between the tracing information block and the inference structured data block to query the tracing information of the target network traffic.
[0011] According to a network traffic source tracing device based on multimodal data identification provided by the present invention, the device also includes: The tracing module is used to obtain keywords of target network traffic through user input after constructing an association body according to the mapping relationship between the tracing information block and the inference structured data block; and to trace the keywords based on the association body to obtain target tracing information.
[0012] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, a network traffic tracing method based on multimodal data identification as described above is implemented.
[0013] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements any of the network traffic tracing methods based on multimodal data identification as described above.
[0014] The present invention also provides a computer program product, including a computer program, which, when executed by a processor, implements any of the network traffic tracing methods based on multimodal data identification as described above.
[0015] The network traffic tracing method and device based on multimodal data identification provided by the present invention extract traffic summary information from multimodal network traffic data, infer multiple data contents, and then use the traffic summary information to mark the tracing information and the inferred structured data respectively, and then establish a mapping relationship between the tracing information block and the inferred structured data block to obtain an associated body, thereby realizing efficient query of traffic tracing information and improving the tracing efficiency of multimodal data traffic in complex scenarios. BRIEF DESCRIPTION OF THE DRAWINGS
[0016] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.
[0017] Figure 1 It is one of the flow charts of the network traffic tracing method based on multimodal data identification provided by the present invention.
[0018] Figure 2 It is a flow chart of the multimodal traffic data reasoning and information block construction method provided by the present invention.
[0019] Figure 3 It is a flowchart of the method for querying traceability information through keywords provided by the present invention.
[0020] Figure 4 This is the second flow chart of the network traffic tracing method based on multimodal data identification provided by the present invention.
[0021] Figure 5 It is a structural schematic diagram of a network traffic tracing device based on multimodal data identification provided by the present invention.
[0022] Figure 6 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION
[0023] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0024] Combine the following Figure 1-Figure 5 The present invention describes a network traffic tracing method and device based on multimodal data identification.
[0025] Figure 1 This is one of the flow charts of the network traffic tracing method based on multimodal data identification provided by the present invention, such as Figure 1 As shown, the method comprises the following steps: Step 110: extract traffic summary information from the multimodal network traffic data; wherein the multimodal network traffic data includes multiple data contents and multiple traceable information.
[0026] In this step, the various data contents include but are not limited to data in various modalities such as text, image, audio, and video.
[0027] In this step, the quintuple method, context semantic analysis method and graph model-based method can be used to extract traffic summary information from multimodal network traffic data.
[0028] For example, methods based on graph models include: representing traffic data as a graph structure, where nodes represent IP addresses or port numbers, and edges represent traffic transmission relationships; the graph model can be used to analyze traffic direction, transmission paths, etc.; or using algorithms such as TextRank to build a similarity matrix based on relevant features of traffic data (such as packet size, transmission time, etc.), and running algorithms such as PageRank to sort the traffic data, thereby extracting important traffic summary information.
[0029] In this embodiment, the various traceable information includes, but is not limited to, data such as flow IP information, User-Agent information in HTTP traffic, Cookie information, and URI information.
[0030] Step 120, infer multiple data contents to obtain inferred structured data, and generate an inferred structured data block with the traffic summary information as the first data header and the inferred structured data as the first data body; generate a traceability information block with the traffic summary information as the second data header and the traceability information as the second data body.
[0031] In this step, various data contents are inferred according to certain relationship templates, and corresponding descriptive texts can be generated. In the field of network security, this descriptive text can help identify potential events, attack types, results, and occurrence times.
[0032] In this embodiment, a large inference model obtained through pre-training can be used to infer the above-mentioned various data contents, which can improve data inference efficiency and accuracy.
[0033] In this embodiment, if the various data contents include long videos or long audios with long playback time, the long videos and long audios can be segmented in a fine-grained manner to optimize the processing efficiency of the large inference model.
[0034] For example, a variety of data contents include pictures, long videos, and long audios; reasoning is performed on a variety of data contents to obtain inferred structured data including: fine-grained segmentation of pictures, long videos, and long audios to obtain at least two of pictures, fine-grained audios, and fine-grained videos; and the pictures, fine-grained audios, and fine-grained video contents are processed based on the inference big model to obtain inferred structured data.
[0035] In this embodiment, the image data among the multiple data contents include an image whose file name is abnormal and contains the words "malware_sample.jpg", and the file is requested multiple times during the transmission process; the video data among the multiple data contents include: a regular teaching video watched by a user, but the video stream is suddenly interrupted at a certain point in time and then restored, and abnormal data packets are transmitted during the interruption; another video file from an unknown source is quickly spread in the local area network through P2P, and the file content is suspected of containing malicious code; the audio data among the multiple data contents include: sensitive information (such as passwords) mentioned in the user's call recording, and there is a sudden drop in audio quality during the call.
[0036] Based on the inference model, the above data content is inferred as follows and the corresponding descriptive text is generated: (1) Image data: An abnormal file name "malware_sample.jpg" may indicate that the file is a malware sample; multiple requests may indicate that an automated script or malware is attempting to download or spread the file.
[0037] (2) Video data: Interruption of video stream may indicate a network attack (such as DDoS) or equipment failure; abnormal data packets may contain attack payloads or information used to detect network vulnerabilities; video files from unknown sources spread through P2P may indicate the presence of worm viruses or malware; file content suspected of containing malicious code may mean that the file is destructive.
[0038] (3) Audio data: Leakage of sensitive information may indicate an insider leak or eavesdropping attack; degradation of audio quality may be caused by a network attack (such as VoIP hijacking).
[0039] (4) Generate descriptive text (i.e., infer structured data): 1. The text content includes: within a specific time range, it is detected that an image file named "malware_sample.jpg" is requested multiple times, which may involve the spread of malware; within the same time period, the teaching video stream watched by the user is suddenly interrupted at A:B and then restored, during which there are abnormal data packets transmitted, which may be subject to network attacks; in the call recording from C:C to D:D, the user mentioned sensitive information, and the audio quality suddenly dropped during the call, which may be subject to monitoring attacks; at E:F, the system detected that a video file from an unknown source was rapidly spread in the local area network through P2P, and the file content is suspected to contain malicious code, which may cause network security incidents; 2. Attack types include: malware spread (image data), network attack (video data interruption), monitoring attack (audio data), worm virus or malware spread (another video data); 3. Results: The malware sample was identified and isolated; the video stream interruption event was recorded, and the abnormal data packet is being further analyzed; the sensitive information leakage event has been reported to the relevant departments, and call encryption measures have been strengthened; the suspected malicious video file has been blocked from spreading, and detailed analysis is being carried out; All events occurred within the target time period.
[0040] In this embodiment, the inference structured data includes multiple different types of data bodies (including descriptive texts corresponding to audio, images, and videos). These different data bodies can be linked to obtain a chain data body, and then the traffic summary information is used as the data header and the chain data body is used as the data body to construct an inference structured data block of a multimodal data stream.
[0041] Similarly, a traceable information block of a multimodal data stream is constructed using the traffic summary information as the data header and a variety of traceable information as the data body.
[0042] Figure 2 is a flow chart of the multimodal traffic data reasoning and information block construction method provided by the present invention. Figure 2 In the illustrated embodiment, a method for inferring multimodal traffic data and constructing information blocks includes: 201, obtaining data traffic from a collection module, completing data packaging according to traffic characteristics, and screening out traffic containing multimodal data, including: pictures, audio and video data; 202, inferring structured data, including a data header and a chain data body; wherein the data header is a traffic information summary of the multimodal data; the chain data body is inferring structured data 1 to inferring structured data n; when the multimodal data traffic is fine-grained, each fine-grained data block generates inferring structured data n through the inference module, and the n inferring structured data blocks are combined as a chain data body with the data header traffic information summary to form an inferring structured data of a multimodal data stream; 203, traffic traceability data, including a data header and a data body; wherein the data header is a traffic information summary of the multimodal data, and the data body is the traffic traceability information.
[0043] In this embodiment, for the same multimodal data flow, the headers of the inferred structured data and the traceable information are both information summaries of the same flow, and the traceability function of the multimodal data is realized through the association relationship of the data body header information.
[0044] Step 130: construct an association according to the mapping relationship between the tracing information block and the inference structured data block to query the tracing information of the target network traffic.
[0045] In this step, the inference structured data block may include: basic communication information (such as source IP address, destination IP address, source port, destination port, protocol type (such as TCP, UDP, etc.), traffic characteristics (traffic size, transmission rate, number of packets, etc.), timestamp (time when network traffic occurs) and session (session start time, end time and session length) and other information.
[0046] In this step, the traceability information is used to identify key information such as the attacker's identity, location, attack path, etc. by analyzing network traffic data; specifically, the traceability information may also include: attacker identity, attack path, attack time, and attack means, etc.
[0047] In this embodiment, the inference structured data block is mapped and associated with the traceability information block, including matching and associating the attacker's identity, location, attack path and other information with the IP address, port number, timestamp and other information in the network traffic. Through this mapping relationship, the attacker's action path and strategy in the network can be more intuitively understood.
[0048] In this embodiment, the target network traffic may be network traffic intercepted through a related traffic interface or traffic data associated with a keyword input according to specific needs of the user.
[0049] In this embodiment, the mapping relationship can also be presented in a visual form; for example, visualization tools such as charts and graphs can be used to display information such as attack paths and traffic characteristics, which helps the security team better understand the threat level and impact scope of the attack and formulate more efficient response strategies.
[0050] The network traffic tracing method based on multimodal data identification provided by the embodiment of the present invention extracts traffic summary information from multimodal network traffic data, infers multiple data contents, and then uses the traffic summary information to mark the tracing information and the inferred structured data respectively, and then establishes a mapping relationship between the tracing information block and the inferred structured data block to obtain an associated body, thereby realizing efficient query of traffic tracing information and improving the tracing efficiency of multimodal data traffic in complex scenarios.
[0051] In some embodiments, extracting traffic summary information from multimodal network traffic data includes: parsing the multimodal network traffic data and extracting quintuple information from the parsed traffic data; and generating traffic summary information based on the quintuple information.
[0052] In this embodiment, the quintuple information includes basic attributes of network traffic, for example, the quintuple information includes source IP address, destination IP address, source port number, destination port number and protocol type, which is used to count and generate traffic information summary.
[0053] Specifically, the steps of extracting quintuple information and generating a traffic information summary include: (1) Data preprocessing: First, parse the traffic records containing multimodal data, such as text logs, binary data packets, and logs in JSON / XML format. Clean the data by removing irrelevant information, such as timestamps (unless needed for time analysis) and redundant fields.
[0054] (2) Extract five-tuple information; Source IP address: record the IP address of the initiator of the traffic; Destination IP address: record the IP address of the receiver of the traffic; Source port number: record the port number of the initiator of the traffic; Destination port number: record the port number of the receiver of the traffic; Protocol type: record the network protocol used (such as TCP, UDP, HTTP, etc.).
[0055] (3) Generate traffic information summary; the summary information includes traffic statistics: calculation of total traffic, average traffic rate, traffic peak, etc.; session statistics: counting the number of unique five-tuple sessions to understand the activity level of different sessions; protocol distribution: counting the frequency of use of different protocols; IP address distribution: counting the frequency of occurrence of source IP and destination IP to identify possible hot spots or abnormal behaviors.
[0056] The network traffic tracing method based on multimodal data identification provided by the embodiment of the present invention extracts quintuple information from the parsed traffic data; generates traffic summary information based on the quintuple information, thereby improving the efficiency of traffic summary extraction and further improving the efficiency of multimodal data parsing.
[0057] In some embodiments, multimodal network traffic data is obtained through the following steps: obtaining the network traffic data required for detecting deep messages based on a data traffic interface; filtering and reorganizing the network traffic data, extracting multiple types of traffic features from the reorganized network traffic data, and generating multimodal network traffic data based on the multiple types of traffic features.
[0058] In this embodiment, the data traffic interface may be deployed on network devices (such as routers and switches) to capture all data packets passing through these devices.
[0059] In this embodiment, the traffic data in the network can be acquired in real time or periodically through a dedicated data traffic interface.
[0060] In this embodiment, in the process of acquiring network traffic data, deep packet inspection technology can be used to parse data packets to identify key information such as application type and user behavior, providing a basis for subsequent traffic feature extraction and pattern recognition.
[0061] In this embodiment, after the original network traffic data is obtained, data filtering is first performed, including removing invalid data packets (such as packets with protocol errors, duplicate packets, etc.) and filtering out data packets that are not related to the detection target. Data filtering can reduce the complexity and computational complexity of subsequent processing.
[0062] In this embodiment, after data filtering, data reorganization is performed (for recombining the filtered data packets into more meaningful traffic units according to certain rules); for example, data packets of the same session can be combined into a session flow, or data packets in the same time period can be combined into a traffic segment; through data reorganization, traffic features can be extracted more conveniently and pattern recognition can be performed.
[0063] In this embodiment, statistical analysis is performed on the reorganized network traffic data to extract statistical features such as message length, transmission delay, and traffic size to reflect the overall situation and behavioral trends of the network traffic. Spectral analysis technology can also be used to perform spectrum conversion on the time domain signal of the network traffic to extract spectral features such as frequency distribution and frequency components, which helps to reveal the periodic behavior and frequency characteristics in the network traffic. Machine learning algorithms can also be used to train and learn network traffic data to automatically extract more advanced feature representations. For example, a convolutional neural network (CNN) can be used to visualize the network traffic and then extract the visual features of the image to more accurately describe the complexity and diversity of the network traffic.
[0064] In this embodiment, after extracting various types of traffic features, these features are fused and integrated to generate multimodal network traffic data; for example, feature fusion and integration are achieved through feature splicing, feature fusion and other technologies, which can fully utilize the advantages of various features and improve the accuracy and reliability of network traffic data.
[0065] The network traffic tracing method based on multimodal data identification provided by the embodiment of the present invention obtains the network traffic data required for detecting deep messages based on the data traffic interface; filters and reorganizes the network traffic data, extracts multiple types of traffic features from the reorganized network traffic data, and generates multimodal network traffic data based on the multiple types of traffic features, thereby improving the quality of the multimodal network traffic data and providing data support for subsequent traffic tracing.
[0066] In some embodiments, after constructing an association body based on the mapping relationship between the traceability information block and the inference structured data block, the network traffic tracing method based on multimodal data identification also includes: obtaining keywords of the target network traffic through user input; tracing the keywords based on the association body to obtain target tracing information.
[0067] In this embodiment, corresponding keywords are obtained through user input, and then the content in the inference structured data of the multimodal traffic is hit by keyword query, and finally the traceable information of the traffic is obtained through the association relationship of the traffic information summary and output as the query result.
[0068] Figure 3 is a flow chart of a method for querying source tracing information by keywords provided by the present invention. Figure 3In the illustrated embodiment, a method for querying traceability information through keywords includes the following steps: Step 301, for processed multimodal traffic data, first perform a content query on the multimodal data based on keywords; Step 302, query the multimodal reasoning structured data content in the keyword hit database, and be able to obtain the traffic information summary of the data header; Step 303, obtain the traceability information of the traffic through the traffic information summary association, and return it as the query result.
[0069] The network traffic tracing method based on multimodal data identification provided by the embodiment of the present invention obtains keywords of the target network traffic through user input; traces the keywords based on the association body to obtain the target tracing information, thereby further improving the tracing efficiency of the multimodal data traffic.
[0070] Figure 4 This is the second flow chart of the network traffic tracing method based on multimodal data identification provided by the present invention. Figure 4 In the illustrated embodiment, a network traffic tracing method for multimodal data identification includes the following steps: step 401, obtaining traffic and grouping, filtering out traffic containing multimodal data; step 402, extracting traffic five-tuples and generating a traffic information summary; step 403, preprocessing the multimodal data and importing it into the inference module to generate structured data describing the multimodal content; step 404, using the traffic information summary to mark the multimodal content structured data; step 405, extracting traceable information from traffic containing multimodal data; step 406, using the traffic information summary to mark the traceable information of the multimodal traffic data; step 407, associating the traceable information of the multimodal traffic with the structured data of the multimodal content through the traffic feature summary; step 408, when retrieving the multimodal data content, the traffic information summary can be associated with the traceability information of the traffic.
[0071] The network traffic tracing device based on multimodal data identification provided by the present invention is described below. The network traffic tracing device based on multimodal data identification described below and the network traffic tracing method based on multimodal data identification described above can be referenced to each other.
[0072] Figure 5 is a schematic diagram of the structure of a network traffic tracing device based on multimodal data identification provided by the present invention, such as Figure 5 As shown, the network traffic tracing device based on multimodal data identification includes: an information extraction module 510, a data block generation module 520 and an association module 530.
[0073] The information extraction module 510 is used to extract traffic summary information from the multimodal network traffic data; wherein the multimodal network traffic data includes multiple data contents and multiple traceable information; The data block generation module 520 is used to infer multiple data contents to obtain inference structured data, and use the traffic summary information as the first data header and the inference structured data as the first data body to generate an inference structured data block; use the traffic summary information as the second data header and the traceability information as the second data body to generate a traceability information block; The association module 530 is used to construct an association body according to the mapping relationship between the tracing information block and the inference structured data block to query the tracing information of the target network traffic.
[0074] The network traffic tracing device based on multimodal data identification provided by the embodiment of the present invention extracts traffic summary information from multimodal network traffic data, infers multiple data contents, and then uses the traffic summary information to mark the tracing information and the inferred structured data respectively, and then establishes a mapping relationship between the tracing information block and the inferred structured data block to obtain an associated body, thereby realizing efficient query of traffic tracing information and improving the tracing efficiency of multimodal data traffic in complex scenarios.
[0075] exist Figure 5 In the illustrated embodiment, the network traffic tracing device based on multimodal data identification also includes: a tracing module 540, which is used to obtain keywords of the target network traffic through user input after constructing an association body according to the mapping relationship between the tracing information block and the inference structured data block; and trace the keywords based on the association body to obtain the target tracing information.
[0076] The network traffic tracing device based on multimodal data identification provided by the embodiment of the present invention obtains keywords of the target network traffic through user input; traces the keywords based on the association body to obtain the target tracing information, thereby further improving the tracing efficiency of the multimodal data traffic.
[0077] Figure 6 is a schematic diagram of the structure of the electronic device provided by the present invention, such as Figure 6As shown, the electronic device may include: a processor 610, a communication interface 620, a memory 630 and a communication bus 640, wherein the processor 610, the communication interface 620 and the memory 630 communicate with each other through the communication bus 640. The processor 610 may call the logic instructions in the memory 630 to execute a network traffic tracing method based on multimodal data identification, the method comprising: extracting traffic summary information from multimodal network traffic data; wherein the multimodal network traffic data includes multiple data contents and multiple traceable information; reasoning the multiple data contents to obtain inference structured data, and generating an inference structured data block with the traffic summary information as the first data header and the inference structured data as the first data body; generating a traceability information block with the traffic summary information as the second data header and the traceability information as the second data body; constructing an association body according to the mapping relationship between the traceability information block and the inference structured data block to query the traceability information of the target network traffic.
[0078] In addition, the logic instructions in the above-mentioned memory 630 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.
[0079] On the other hand, the present invention also provides a computer program product, which includes a computer program, and the computer program can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the network traffic tracing method based on multimodal data identification provided by the above-mentioned methods, and the method includes: extracting traffic summary information from multimodal network traffic data; wherein the multimodal network traffic data includes multiple data contents and multiple traceable information; reasoning the multiple data contents to obtain inferred structured data, and using the traffic summary information as the first data header and the inferred structured data as the first data body to generate an inferred structured data block; using the traffic summary information as the second data header and the traceability information as the second data body to generate a traceability information block; constructing an association body according to the mapping relationship between the traceability information block and the inferred structured data block to query the traceability information of the target network traffic.
[0080] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to execute the network traffic tracing method based on multimodal data identification provided by the above-mentioned methods, the method comprising: extracting traffic summary information from multimodal network traffic data; wherein the multimodal network traffic data comprises a variety of data contents and a variety of traceable information; reasoning the multiple data contents to obtain inferred structured data, and generating an inferred structured data block with the traffic summary information as the first data header and the inferred structured data as the first data body; generating a traceability information block with the traffic summary information as the second data header and the traceability information as the second data body; and constructing an association body according to the mapping relationship between the traceability information block and the inferred structured data block to query the traceability information of the target network traffic.
[0081] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.
[0082] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.
[0083] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.
Claims
1. A network traffic tracing method based on multimodal data recognition, characterized in that: include: Extracting traffic summary information from multimodal network traffic data; wherein the multimodal network traffic data includes multiple data contents and multiple traceable information; Inferring the multiple data contents to obtain inference structured data, and generating an inference structured data block with the traffic summary information as the first data header and the inference structured data as the first data body; generating a traceability information block with the traffic summary information as the second data header and the traceability information as the second data body; An association is constructed according to the mapping relationship between the tracing information block and the inference structured data block to query the tracing information of the target network traffic.
2. The network traffic tracing method based on multimodal data identification according to claim 1 is characterized in that: The extracting of traffic summary information from the multimodal network traffic data includes: Parsing the multimodal network traffic data and extracting quintuple information from the parsed traffic data; The traffic summary information is generated according to the quintuple information.
3. The network traffic tracing method based on multimodal data identification according to claim 1 is characterized in that: The multiple data contents include pictures, long videos and long audios; The reasoning on the multiple data contents to obtain the inferred structured data includes: The picture, the long video and the long audio are respectively segmented into fine-grained segments to obtain at least two of the picture, the fine-grained audio and the fine-grained video. The picture, fine-grained audio, and fine-grained video content are processed based on the inference big model to obtain the inference structured data.
4. The network traffic tracing method based on multimodal data identification according to claim 1 is characterized in that: The multimodal network traffic data is obtained by the following steps: Obtain network traffic data required for deep message detection based on the data traffic interface; The network traffic data is filtered and reorganized, multiple types of traffic features are extracted from the reorganized network traffic data, and the multimodal network traffic data is generated according to the multiple types of traffic features.
5. The network traffic tracing method based on multimodal data identification according to claim 1 is characterized in that: After constructing the association according to the mapping relationship between the traceability information block and the inference structured data block, the method further includes: Obtain keywords of target network traffic through user input; The keyword is traced based on the association to obtain target traceability information.
6. A network traffic tracing device based on multimodal data recognition, characterized in that: include: An information extraction module, used to extract traffic summary information from multimodal network traffic data; wherein the multimodal network traffic data includes multiple data contents and multiple traceable information; A data block generation module is used to infer the multiple data contents to obtain inference structured data, and to generate an inference structured data block with the traffic summary information as the first data header and the inference structured data as the first data body; and to generate a traceability information block with the traffic summary information as the second data header and the traceability information as the second data body; The association module is used to construct an association body according to the mapping relationship between the tracing information block and the inference structured data block to query the tracing information of the target network traffic.
7. The network traffic tracing device based on multimodal data identification according to claim 6 is characterized in that: The device also includes: The tracing module is used to obtain keywords of target network traffic through user input after constructing an association body according to the mapping relationship between the tracing information block and the inference structured data block; and to trace the keywords based on the association body to obtain target tracing information.
8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that: When the processor executes the computer program, the network traffic tracing method based on multimodal data identification as described in any one of claims 1 to 5 is implemented.
9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the network traffic tracing method based on multimodal data identification as described in any one of claims 1 to 5 is implemented.
10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the network traffic tracing method based on multimodal data identification as described in any one of claims 1 to 5 is implemented.