Method, device, equipment and storage medium for quickly finding and deleting published files

Abstract

The embodiment of the present application provides a method, apparatus, device and storage medium for quickly searching and deleting published files, the method comprising: using a file transmission module to receive a target file, sending the received target file to a content distribution network module, and sending information of the target file to a message server; using a file parsing module to obtain information of the target file on the message server, obtain the target file, parse the target file, and push the parsing result to an index module; using the index module to query files that meet preset conditions according to the parsing result, and sending them to the file transmission module as target files to be deleted; using the file transmission module to delete the target files to be deleted in the content distribution network module, adding a message notification mechanism, parsing the target file in real time, retrieving the parsing result, performing subsequent processing and deletion, and being able to comprehensively query and delete pages containing illegal content.

Description

Technical Field

[0001] The present application relates to the technical field of website operation management, and in particular to a method, device, equipment and storage medium for quickly searching and deleting published files. Background Art

[0002] In the daily operation and management of the website, it is often necessary to quickly delete some pages, such as those containing negative content such as immoral artists. Figure 1 In general daily operations, the content management and publishing system is used to query whether the page contains illegal content, and then the content management and publishing system is used to issue instructions to delete the page containing illegal content. The file transfer system is used to operate and delete the page containing illegal content on the source site.

[0003] However, in daily operations, when checking whether a page contains illegal content through the content management and publishing system, some pages are often missed or deleted, such as some special pages or included pages that contain illegal content, so the records of pages containing illegal content in the content management and publishing system are incomplete. At the same time, the website has multiple domain names and channels, and deletion requires coordination with multiple operation teams, which makes the response slow. Summary of the invention

[0004] Embodiments of the present application provide a method, apparatus, device, and storage medium for quickly finding and deleting published files.

[0005] A first aspect of an embodiment of the present application provides a method for quickly searching and deleting a published file, comprising:

[0006] Receive the target file by using the file transfer module, send the received target file to the content distribution network module, and send the information of the target file to the message server;

[0007] Use the file parsing module to obtain the information of the target file on the message server, obtain the target file, parse the target file, and push the parsed file title, time and text information to the index module;

[0008] The index module is used to search for files that meet the preset conditions according to the file title, time and text information, and the files are sent to the file transfer module as the target files to be deleted;

[0009] The file transfer module is used to delete the target file to be deleted in the content distribution network module.

[0010] In an optional embodiment of the present application, a file parsing module is used to obtain information of a target file on a message server, obtain the target file, parse the target file, and push the parsed file title, time, and text information to an indexing module, including:

[0011] Get the file type of the target file;

[0012] According to the file type of the target file, determine whether the target file needs to be preprocessed;

[0013] When the target file needs to be preprocessed, the target file is preprocessed, and then the preprocessed target file is parsed;

[0014] Without preprocessing the target file, the target file is directly parsed;

[0015] Push the parsed file title, time and text information to the index module.

[0016] In an optional embodiment of the present application, determining whether the target file needs to be preprocessed according to the file type of the target file includes:

[0017] When the file type of the target file is a text file, there is no need to preprocess the target file;

[0018] When the file type of the target file is a JSON file or an HTML file, the target file needs to be preprocessed.

[0019] In an optional embodiment of the present application, when the target file needs to be preprocessed, the target file is preprocessed, including:

[0020] When the file type of the target file is a JSON file, delete the JSON tag in the JSON file;

[0021] When the file type of the target file is an HTML file, and the HTML file contains at least one of audio and video, obtain the globally unique identifiers of the audio and video, obtain the identification text of the audio and video in the HTML file through the globally unique identifiers of the audio and video, and then use the identification text as the HTML file to extract the largest body text block of the HTML file; when the HTML file does not contain at least one of audio and video, extract the largest body text block of the HTML file.

[0022] In an optional embodiment of the present application, extracting the largest body text block of the HTML file includes:

[0023] Parse the tags in the HTML file into a DOM tree and obtain the total number of tags in the HTML file and the location information of each tag;

[0024] According to the number of line break tags and paragraph tags in each node of the HTML file, all nodes are sorted to get the top N nodes;

[0025] For each of the top N nodes, perform secondary sorting based on the number of links, the number of Chinese punctuation marks, the total number of words and the number of words with links, the total number of tags in the HTML file, and the position information of each tag, and take the first node as the largest text block;

[0026] Move forward from the node position of the largest text block, extract the title of the HTML file, extract the release time of the HTML file through a regular expression, and if the extraction is successful, use the text of the first node as the text of the HTML file; if the extraction is unsuccessful, use the second node in the secondary sorting as the largest text block, and extract the title and release time of the HTML file until the extraction is successful;

[0027] When the first node is taken as the largest body text block, if the body of the first node cannot be extracted or the number of extracted body words is less than a preset threshold, the html tag in the HTML file is directly deleted, and the HTML file with the html tag deleted is taken as the body of the HTML file.

[0028] In an optional embodiment of the present application, all nodes are sorted according to the number of line break tags and paragraph tags in each node on the DOM tree to obtain the top N nodes, including:

[0029] The first weight value of each node is determined by the following expression:

[0030] W1 = brCount + 2 pCount

[0031] Where W1 is the first weight value of each node, brCount is the number of line break tags of each node, and pCount is the number of paragraph tags of each node.

[0032] All nodes are sorted from largest to smallest according to their first weight values ​​to obtain the top N nodes.

[0033] In an optional embodiment of the present application, for each of the top N nodes, a secondary sorting is performed on all the top N nodes according to the number of links, the number of Chinese punctuation marks, the total number of words and the number of words with links of the current node, the total number of tags of the HTML file, and the position information of each tag, including:

[0034] The second weight value of each node is determined by the following expression:

[0035] W2 = (WN-LWL) × dl × f

[0036] Wherein, W2 is the second weight value of each node, WN is the total number of words of each node, LWL is the number of linked words of each node, dl is the location information of each node, and f is the weight factor of each node.

[0037] According to the second weight values ​​of all the nodes in the top N, sort them again from large to small.

[0038] Among them, the position information of each node is obtained by the following expression:

[0039]

[0040] Among them, dl is the location information of each node, NodeCount and NodePos are the total number of tags in the HTML file and the location information of each tag, respectively.

[0041] The weight factor of each node is obtained by the following expression:

[0042]

[0043] Among them, f is the weight factor of each node, dLinkPower is the number of links of each node, and nCDotNum is the number of Chinese punctuation marks of each node.

[0044] A second aspect of the embodiment of the present application provides a device for quickly searching and deleting published files, including:

[0045] A file transmission module, used for receiving a target file, sending the received target file to the content distribution network module, sending information of the target file to the message server, and also for deleting the target file to be deleted in the content distribution network module;

[0046] A message server, used to provide information about target files;

[0047] The file parsing module is used to obtain the information of the target file on the message server, obtain the target file, parse the target file, and push the parsed file title, time and text information to the indexing module;

[0048] The index module is used to search for files that meet preset conditions according to the file title, time and text information, and send them to the file transfer module as target files to be deleted.

[0049] According to a third aspect of an embodiment of the present application, a computer device is provided, including: a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the steps of any of the above methods for quickly finding and deleting published files are implemented.

[0050] A fourth aspect of an embodiment of the present application provides a computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, the steps of any of the above methods for quickly finding and deleting published files are implemented.

[0051] The above technical solution provided by the embodiment of the present application has at least some or all of the following advantages compared with the prior art:

[0052] The method for quickly finding and deleting published files described in the embodiment of the present application comprises the following steps: receiving a target file by a file transfer module, sending the received target file to a content distribution network module, and sending information of the target file to a message server; obtaining information of the target file on the message server by a file parsing module, obtaining the target file, parsing the target file, and pushing the parsed file title, time and text information to an index module; searching for files meeting preset conditions according to the file title, time and text information in the index module, and sending the files as target files to be deleted to the file transfer module; deleting the target files to be deleted in the content distribution network module by the file transfer module, adding a message notification mechanism between the file transfer module and the content distribution network module, parsing the target file sent from the file transfer module to the content distribution network module, and sending the parsing result to the index module; performing subsequent processing and deletion by retrieving the parsing result of the index module, and thus being able to comprehensively query and delete pages containing illegal content. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0054] Figure 1 A flowchart of a method for quickly finding and deleting published files provided for related technologies;

[0055] Figure 2 A flowchart of a method for quickly finding and deleting published files provided by an embodiment of the present application;

[0056] Figure 3 A flowchart of a method for quickly searching and deleting published files provided by another embodiment of the present application;

[0057] Figure 4A schematic diagram of a query interface of an index module provided in one embodiment of the present application;

[0058] Figure 5 A text schematic diagram of an HTML file obtained through globally unique identifiers of audio and video provided by one embodiment of the present application;

[0059] Figure 6 A schematic diagram of the structure of a device for quickly searching and deleting published files provided in one embodiment of the present application;

[0060] Figure 7 A schematic diagram of the computer device structure provided for one embodiment of the present application. DETAILED DESCRIPTION

[0061] In order to make the technical solutions and advantages in the embodiments of the present application more clearly understood, the exemplary embodiments of the present application are further described in detail below in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present application, rather than an exhaustive list of all the embodiments. It should be noted that the embodiments in the present application and the features in the embodiments can be combined with each other without conflict.

[0062] See also Figure 2 The method for quickly searching and deleting published files provided in the embodiment of the present application includes the following steps 100 to 400:

[0063] Step 100: Receive a target file using a file transfer module, send the received target file to a content distribution network module, and send information about the target file to a message server;

[0064] Step 200: Use the file parsing module to obtain information about the target file on the message server, obtain the target file, parse the target file, and push the parsed file title, time, and text information to the indexing module;

[0065] Step 300: Use the index module to search for files that meet preset conditions based on the file title, time and text information, and send them to the file transfer module as target files to be deleted;

[0066] Step 400: Utilize the file transfer module to delete the target file to be deleted in the content distribution network module.

[0067] See also Figure 3All files are uploaded to the source site of the content management and publishing module via the file transfer module. A message notification mechanism is added to the file transfer module to notify the message server of all HTML files (HTML files are files written in Hypertext Markup Language), JSON (JavaScript Object Notation) files, txt files (plain text files), etc. uploaded to the source site. The file parsing module obtains the message on the message server, parses all uploaded files, and pushes the parsed file title, time, and text information to the index module.

[0068] In an optional embodiment of the present application, in step 200, the file parsing module is used to obtain information of the target file on the message server, obtain the target file, parse the target file, and push the parsed file title, time and text information to the index module, including:

[0069] Get the file type of the target file;

[0070] According to the file type of the target file, determine whether the target file needs to be preprocessed;

[0071] When the target file needs to be preprocessed, the target file is preprocessed, and then the preprocessed target file is parsed;

[0072] Without preprocessing the target file, the target file is directly parsed;

[0073] Push the parsed file title, time and text information to the index module.

[0074] In an optional embodiment of the present application, determining whether the target file needs to be preprocessed according to the file type of the target file includes:

[0075] When the file type of the target file is a text file, there is no need to preprocess the target file;

[0076] When the file type of the target file is a JSON file or an HTML file, the target file needs to be preprocessed.

[0077] In an optional embodiment of the present application, when the target file needs to be preprocessed, the target file is preprocessed, including:

[0078] When the file type of the target file is a JSON file, delete the JSON tag in the JSON file;

[0079] When the file type of the target file is an HTML file, and the HTML file contains at least one of audio and video, obtain the globally unique identifiers of the audio and video, obtain the identification text of the audio and video in the HTML file through the globally unique identifiers of the audio and video, and then use the identification text as the HTML file to extract the largest body text block of the HTML file; when the HTML file does not contain at least one of audio and video, extract the largest body text block of the HTML file.

[0080] See also Figure 3 , when the HTML file contains at least one of audio and video, the globally unique identifiers (guid in the figure) of the audio and video can be extracted according to the playback information of the audio and video, and then the AI ​​middle station is called through the globally unique identifier to obtain the text description of the audio and video files (including text information such as characters and locations), such as Figure 4 shown.

[0081] In an optional embodiment of the present application, extracting the largest body text block of the HTML file includes:

[0082] Parse the tags in the HTML file into a DOM (Document Object Model) tree, and obtain the total number of tags in the HTML file and the location information of each tag;

[0083] According to the number of line break tags and paragraph tags in each node of the HTML file, all nodes are sorted to get the top N nodes;

[0084] For each of the top N nodes, perform secondary sorting based on the number of links, the number of Chinese punctuation marks, the total number of words and the number of words with links, the total number of tags in the HTML file, and the position information of each tag, and take the first node as the largest text block;

[0085] Move forward from the node position of the largest text block, extract the title of the HTML file, extract the release time of the HTML file through a regular expression, and if the extraction is successful, use the text of the first node as the text of the HTML file; if the extraction is unsuccessful, use the second node in the secondary sorting as the largest text block, and extract the title and release time of the HTML file until the extraction is successful;

[0086] When the first node is taken as the largest body text block, if the body of the first node cannot be extracted or the number of extracted body words is less than a preset threshold, the html tag in the HTML file is directly deleted, and the HTML file with the html tag deleted is taken as the body of the HTML file.

[0087] In an optional embodiment of the present application, all nodes are sorted according to the number of line break tags and paragraph tags in each node on the DOM tree to obtain the top N nodes, including:

[0088] The first weight value of each node is determined by the following expression:

[0089] W1 = brCount + 2 pCount

[0090] Where W1 is the first weight value of each node, brCount is the number of line break tags of each node, and pCount is the number of paragraph tags of each node.

[0091] All nodes are sorted from largest to smallest according to their first weight values ​​to obtain the top N nodes.

[0092] In an optional embodiment of the present application, for each of the top N nodes, a secondary sorting is performed on all the top N nodes according to the number of links, the number of Chinese punctuation marks, the total number of words and the number of words with links of the current node, the total number of tags of the HTML file, and the position information of each tag, including:

[0093] The second weight value of each node is determined by the following expression:

[0094] W2 = (WN-LWL) × dl × f

[0095] Wherein, W2 is the second weight value of each node, WN is the total number of words of each node, LWL is the number of linked words of each node, dl is the location information of each node, and f is the weight factor of each node.

[0096] According to the second weight values ​​of all the nodes in the top N, sort them again from large to small.

[0097] Among them, the position information of each node is obtained by the following expression:

[0098]

[0099] Among them, dl is the location information of each node, NodeCount and NodePos are the total number of tags in the HTML file and the location information of each tag, respectively.

[0100] The weight factor of each node is obtained by the following expression:

[0101]

[0102] Among them, f is the weight factor of each node, dLinkPower is the number of links of each node, and nCDotNum is the number of Chinese punctuation marks of each node.

[0103] See also Figure 5 In step 300, the index module is used to search for files that meet the preset conditions according to the file title, time and text information as the target files to be deleted, including:

[0104] A search is performed based on a keyword from at least one of the title, time and text information, and after a file with the keyword is retrieved, the file with the keyword is used as a target file to be deleted, wherein the keyword may be a negative field.

[0105] In an optional embodiment of the present application, in step 300, using the index module to search for files that meet preset conditions according to the file title, time and text information as target files to be deleted includes:

[0106] Input the file title, time and text information of the specified file into the pre-trained file evaluation model;

[0107] When the evaluation result output by the model is that the specified file is a target file to be deleted, the specified file is used as a target file to be deleted. In an optional embodiment of the present application, the file evaluation model can be a multimodal large model, and microservices are established through the multimodal large model to form a file evaluation system based on the multimodal large model. The file evaluation system architecture based on the multimodal large model includes hardware, software, databases and functional modules. The system hardware is a high-computing-power server. The system database includes a text database and an image database that are uploaded to the system for review. The system functional modules include a system data conversion module, a content review algorithm encapsulation module, a system output module, an unstructured data processing module, a multimodal data integration module, a content analysis module, a content anomaly quantification module and an intelligent control module.

[0108] The data conversion module is used to determine the type of data input by the user into the system, determine the scene category from the input data, and after preprocessing the data, combine it with different questioning methods for each scene, and then transmit it to the multimodal large model for reasoning;

[0109] The unstructured data processing module is used to perform semantic analysis on unstructured data such as news graphics, audio, and video using a multimodal large model, generate a semantic association graph, extract the core context and information relevance in text and images through in-depth analysis, and construct a preliminary review data set;

[0110] The multimodal data integration module is used to integrate the graphic data, audio data and video data in a unified context, convert them into a graphic-text association vector space, and transmit them to the content analysis module for in-depth analysis of content context and emotion;

[0111] The content analysis module is used to preliminarily determine the degree of matching between the text and the image content based on the semantic association graph generated by the unstructured data processing module, generate a context factor Qycs, and analyze the potential sensitive content in each modal data to generate a sensitivity factor Gmyz; then, based on the context factor Qycs and the sensitivity factor Gmyz, perform hierarchical analysis on the potential emotional factors, social sensitive information and potential misleading factors in the image and text content to generate a multimodal association factor Mmgyz;

[0112] The content anomaly quantification module calculates the audit weight value Hqwz based on the situational factor Qycs, the sensitivity factor Gmyz and the multimodal association factor Mmgyz; then obtains the system audit risk factor Rxz by calculation and associates it with the audit weight value Hqwz, thereby generating the final content anomaly index Cxzs;

[0113] Among them, the calculation formula of the audit weight value Hqwz is:

[0114] ;

[0115] Among them, b1, b2 and b3 represent the preset weight coefficients of the situational factor Qycs, the sensitive factor Gmyz and the multimodal association factor Mmgyz respectively, and B represents the audit correction coefficient;

[0116] The intelligent control module is used to set the compliance threshold T and the audit risk threshold R, and compare the audit weight value Hqwz with the compliance threshold T to determine whether the current content meets the audit standards, and compare the audit risk threshold R with the content anomaly index Cxzs to obtain the final audit result, and selectively trigger manual review or automatic release control plans based on the audit results.

[0117] In an optional embodiment of the present application, the unstructured data processing module is first used to receive the unstructured data of news graphics, audio and video to be reviewed, and perform format standardization processing on the unstructured data, including image resolution adjustment, text language standardization processing and audio and video transcoding;

[0118] Then, the pre-trained network of the multimodal large model is called to extract multi-level features of news pictures, audio and video, including visual features, semantic features and emotional features, and the extracted multi-level features are normalized to generate a unimodal feature matrix; based on the unimodal feature matrix, the multi-level features of news pictures, audio and video are mapped in the multimodal association vector space, and a cross-modal semantic association graph is generated through the semantic embedding mechanism of the multimodal large model;

[0119] Secondly, we use the cross-modal semantic association graph to conduct a detailed analysis of the deep-level association between text and images, extract the core contextual elements in the text and images, and simultaneously analyze the emotional state and expression trends in the audio and video, so as to identify potential sensitive information, misleading information and social influencing factors in the news content in real time, and form multi-modal association semantic indicators;

[0120] Finally, a preliminary review dataset is generated.

[0121] In an optional embodiment of the present application, the multimodal data integration module is used to receive the preliminary review data set generated by the unstructured data processing module, and perform uniform alignment processing of the timing and content of the graphic data, audio data and video data therein through a frame synchronization algorithm;

[0122] Then, a multimodal feature mapping algorithm is applied to fuse the context, emotion and semantic features of text, audio and video into a multimodal feature vector, and a correlation matrix is ​​generated based on the correlation between each modality to perform hierarchical mapping of information density, context relevance and semantic consistency.

[0123] Subsequently, the image-text association vector space is constructed based on the association matrix, and the multimodal data is quantified in a unified semantic space to generate an integrated semantic association representation vector. At the same time, the semantic deviation caused by modality differences is corrected through the context adjustment algorithm.

[0124] Finally, the integrated image-text association vector space is transmitted to the content analysis module.

[0125] In an optional embodiment of the present application, the content analysis module includes a preliminary discrimination unit, a multi-layer sensitivity analysis unit and a correlation factor generation unit;

[0126] The preliminary discrimination unit identifies semantic deviation related data and content matching related data by analyzing the semantic similarity and context consistency between the image and the text; extracts the semantic similarity Ycy, context consistency Sjy, context association deviation Qjl and image and text emotion consistency Twq from the semantic deviation related data and the content matching related data and performs dimensionless processing, and then calculates the context factor Qycs through the following formula:

[0127] ;

[0128] The preset situation threshold Q is compared with the situation factor Qycs to evaluate the situational relevance of text and images in the audit scenario. The specific comparison and evaluation contents are as follows:

[0129] If the context factor Qycs ≥ the context threshold Q, it indicates that the contexts of the text and the image match, and the semantics, emotions, and contexts of the two are consistent. The review is passed and marked as "normal context association";

[0130] If the context factor Qycs < context threshold Q, it indicates that the context between the text and the image does not match, and the semantics, emotions and context of the two are inconsistent. The review will fail and further analysis of the specific reasons for the deviation will be conducted, including potential sensitive content.

[0131] In an optional embodiment of the present application, the multi-layer sensitivity analysis unit is used to further perform in-depth analysis on the potential sensitive content of the graphic data, including analyzing the emotion-related data, social sensitivity-related data and potential misleading-related data in the text and image, and constructing a sensitive content data set after summarizing and dimensionlessly processing the emotion-related data, social sensitivity-related data and potential misleading-related data in the analyzed text and image; extracting the sensitive content data set and generating the sensitivity factor Gmyz through the following formula:

[0132] ;

[0133] Where Qmq represents the emotion intensity coefficient in the sensitive content data set, Mgc represents the sensitive word index in the sensitive content data set, and Xwg represents the possibility of information misleading in the sensitive content data set.

[0134] In an optional embodiment of the present application, the correlation factor generation unit is used to quantify the degree of correlation between emotion, context and sensitive content based on the context factor Qycs and the sensitivity factor Gmyz by fusing multi-level information in the image-text correlation vector space, and generate a multimodal correlation factor Mmgyz by calculating through the following formula;

[0135]

[0136] In an optional embodiment of the present application, the content anomaly quantification module includes an audit calculation unit and an anomaly index acquisition unit;

[0137] The content evaluation unit is used to construct a system audit risk factor Rxz, obtain the semantic deviation degree Ycy and the content consistency index Nry by extracting semantic deviation related data and content matching related data, and obtain the high sensitivity trigger rate Gmg by extracting the sensitive content data set, and calculate the system audit risk factor Rxz in combination with the following formula:

[0138]

[0139] In an optional embodiment of the present application, the abnormality index obtaining unit is used to calculate and obtain the final content abnormality index Cxzs, and its specific calculation formula is as follows:

[0140]

[0141] In an optional embodiment of the present application, the intelligent control module compares and evaluates the audit weight value Hqwz with the compliance threshold T, and compares and evaluates the audit risk threshold R with the content anomaly index Cxzs, and specifically generates the following evaluation content:

[0142] Compliance standards comparison:

[0143] If the audit weight value Hqwz ≥ the compliance threshold T, it means that the content meets the audit standards and the system marks it as "compliant".

[0144] If the audit weight value Hqwz < the compliance threshold T, it means that the content does not meet the audit standards and is insufficiently compliant, requiring further review or adjustment.

[0145] Abnormal risk assessment:

[0146] If the content anomaly index Cxzs ≥ the audit risk threshold R, it indicates that the content has an abnormal risk, and the system generates an "abnormal content" mark, triggering the manual review process;

[0147] If the content anomaly index Cxzs < the audit risk threshold R, it means that there is no abnormal risk in the content, and the system generates a "normal content" mark, and then enters the automatic publishing process;

[0148] When the evaluation results of the audit weight value Hqwz and the content anomaly index Cxzs are marked as "compliant" and "normal content" at the same time, the content will be automatically marked as "passed" and published directly;

[0149] When the evaluation result of the audit weight value Hqwz and the content anomaly index Cxzs is that the content does not meet the audit standards or has abnormal risks, the content will be automatically marked as "pending review" and the selective review process will be triggered.

[0150] It is used to set the compliance threshold T and the audit risk threshold R, and to determine whether the current content meets the audit standards, obtain the final audit results, and selectively trigger manual review or automatic release control plans based on the audit results.

[0151] In an optional embodiment of the present application, a content audit algorithm encapsulation module is used to encapsulate all algorithms in the system.

[0152] In an optional embodiment of the present application, the system output module is used to extract key information of the results obtained based on the large model and output the final content review category.

[0153] In an optional embodiment of the present application, the first sub-model of the file evaluation model is used to determine whether a file is a file to be deleted based on the file title, time and text information of the specified file.

[0154] In an optional embodiment of the present application, the first sub-model of the file evaluation model is trained by the following steps:

[0155] The first sub-model of the file evaluation model is obtained by taking the file title, time and text information of the historical file as input and whether the file is a file to be deleted as output for training.

[0156] In an optional embodiment of the present application, the second sub-model of the file evaluation model is used to receive and obtain the review result of the comment data of the specified video based on the specified video and its corresponding comment data.

[0157] In an optional embodiment of the present application, the second sub-model of the file evaluation model is trained by the following steps:

[0158] Get the target video and its corresponding comment data;

[0159] Based on the correlation between the target video and its corresponding comment data, the comment data corresponding to the target video is annotated to construct a comment dataset corresponding to the target video;

[0160] The target video and its corresponding comment dataset are used as training sets, and the preset multimodal model is fine-tuned to obtain the second sub-model of the trained file evaluation model.

[0161] In an optional embodiment of the present application, the tagging of the comment data corresponding to the target video based on the correlation between the target video and the comment data corresponding to the target video includes:

[0162] Extract key information and sentiment information from the comment data corresponding to the target video, and preliminarily determine whether the comment data is a negative comment based on the sentiment information;

[0163] If it is preliminarily determined that the comment data is a negative comment, marking the comment data as a negative comment;

[0164] In the case where it is preliminarily determined that the comment data is a positive comment, further determining whether the comment data is a negative comment based on the correlation between the target video and its corresponding comment data;

[0165] If it is further determined that the comment data is a negative comment, marking the comment data as a negative comment;

[0166] When it is further determined that the comment data is a positive comment, the comment data is marked as a positive comment.

[0167] In an optional embodiment of the present application, further determining whether the comment data is a negative comment based on the correlation between the target video and its corresponding comment data includes:

[0168] Extract key information and sentiment information from the target video, and calculate the similarity between the key information and sentiment information in the target video and its corresponding comment data;

[0169] Determine the degree of conflict between the target video and the text content in the corresponding comment data;

[0170] Determine the conflict score between the target video and its corresponding comment data according to the similarity between the key information and emotional information in the target video and its corresponding comment data and the degree of conflict between the text contents;

[0171] The review data whose conflict score exceeds a preset conflict score threshold is determined as a negative review.

[0172] In an optional embodiment of the present application, the similarity between the key information and the sentiment information in the target video and its corresponding comment data is calculated by the following expression:

[0173]

[0174] Among them, A is the first text vector composed of key information and emotional information in the target video, B is the second text vector composed of key information and emotional information in the comment data corresponding to the target video, a i is the i-th vector element in the first text vector, b i is the i-th vector element in the second text vector, and w1 and w2 are both weight factors.

[0175] In an optional embodiment of the present application, constructing a comment dataset corresponding to the target video includes:

[0176] Each comment data corresponding to the target video and the annotation information of each comment data are collected into a comment data set corresponding to the target video.

[0177] In an optional embodiment of the present application, the target video and its corresponding comment data set are used as training sets to fine-tune the preset multimodal model to obtain a trained first multimodal model, including:

[0178] The video frame of the target video and each comment data are taken as input, and the annotation information of the current comment data is taken as output. The network structure used to process the video frames in the preset multimodal model is trained using the lora fine-tuning method of supervised tasks, and the network structure used to process the comment data is trained using the qlora fine-tuning method of supervised tasks to obtain the first multimodal model after training.

[0179] In an optional embodiment of the present application, inputting the specified video and its corresponding comment data into the first multimodal model to obtain the review result of the comment data of the specified video includes:

[0180] Inputting a video frame of a specified video into a network structure for processing the video frame in a first multimodal model to obtain a video frame feature;

[0181] Inputting comment data corresponding to the specified video into a network structure for processing comment data in the first multimodal model to obtain comment data features;

[0182] The video frame features and the comment data features are input into the fusion network structure in the first multimodal model to obtain the review result of the comment data of the designated video.

[0183] In an optional embodiment of the present application, the third sub-model of the file evaluation model is used to receive and determine the target application system failure scenario according to the target application system failure graph.

[0184] In an optional embodiment of the present application, the target application system failure scenario is used to determine the query prompt words of the target application system failure graph, and the target application system failure graph text is extracted according to the query prompt words, so as to decompose the target application system failure text description and the target application system failure graph text into multiple subtasks based on the intelligent agent and the large language model, and determine the target application system failure factors according to the information returned by the subtasks.

[0185] In an optional embodiment of the present application, the third sub-model of the document evaluation model is trained by the following steps:

[0186] Collect historical data of application system failure graphs under each application system failure scenario, wherein the application system failures include video playback errors, sequence errors, data missing, and content errors;

[0187] The application system failure diagram in each application system failure scenario is taken as input, and the application system failure scenario type is taken as output to train the neural network model and obtain the third sub-model of the file evaluation model.

[0188] In an optional embodiment of the present application, the target application system failure scenario is used to determine a query prompt word of the target application system failure graph, including:

[0189] Determine the query prompt word corresponding to the target application system failure scenario based on the correspondence between the preset application system failure scenario and the query prompt word;

[0190] The query prompt word is used as the query prompt word of the target application system fault diagram.

[0191] In an optional embodiment of the present application, the step of extracting target application system fault graph text according to the query prompt word includes:

[0192] Inputting the query prompt words and the target application system fault diagram into the preset multi-modal large model;

[0193] The text extracted from the target application system fault diagram according to the query prompt words is output from the preset multimodal large model as the target application system fault diagram text.

[0194] In an optional embodiment of the present application, the target application system fault text description and the target application system fault graph text are decomposed into multiple subtasks based on the agent and the large language model, and the target application system fault factor is determined according to the information returned by the subtasks, including:

[0195] Inputting the target application system fault text description and the target application system fault graph text into the scheduling agent;

[0196] The scheduling agent determines the fault handling state according to the target application system fault text description and the target application system fault graph text, and sends the target application system fault text description and the target application system fault graph text to the task decomposition agent if the fault is not handled;

[0197] The task decomposition agent decomposes the fault into multiple subtasks according to the preset prompt word template based on the functions of the search engine, interface and database in the preset knowledge base, and sends the multiple subtasks to the task execution agent, wherein each subtask corresponds to at least one of the search engine, interface and database in the preset knowledge base;

[0198] The task execution agent generates executable programs for each subtask according to the prompt word template by using the preset large language model;

[0199] The task execution agent executes each executable program using at least one of the search engine, interface and database according to the corresponding relationship between each executable program and the search engine, interface and database, and uses the execution result of each executable program as the failure factor of the target application system.

[0200] In an optional embodiment of the present application, before taking the execution result of each executable program as a target application system failure factor, the method further includes:

[0201] Send the execution result of each executable program to the task decomposition agent;

[0202] The task decomposition agent determines whether the execution result of each executable program meets the preset conditions. If the preset conditions are not met, the preset prompt word template is adjusted, and the fault is re-decomposed into multiple subtasks according to the adjusted prompt word template to re-execute multiple executable programs until the execution result of each executable program meets the preset conditions. In an optional embodiment of the present application, the method further includes:

[0203] When the system is initialized, the file parsing module fetches all files on the source station of the content distribution network model for parsing;

[0204] When adding, deleting and modifying files in the file transfer model, the file parsing module is notified through the message server, so that the indexing module automatically synchronizes the relevant content on the source station of the content distribution network model.

[0205] In the present application, when the system is initialized, the file parsing module captures all files on the source station of the content distribution network model for parsing; when performing file addition, deletion and modification operations in the file transfer model, the file parsing module is notified through the message server. At the same time, the index module includes information in text and audio and video. Therefore, the files in the index module are comprehensive and updated in real time, so that when the index module queries files with negative content, pages with negative content that need to be deleted will not be missed.

[0206] join Figure 3 , below, taking keyword query for files with negative content as an application scenario, the application process of the method for quickly finding and deleting published files of this application includes editing daily operation process, data preprocessing process and data deletion process:

[0207] 1. Edit daily operation procedures

[0208] Edit and log in to the content management and publishing module, operate the content management and publishing module, publish files, and upload the files to the content distribution network module source station via the content management and publishing module and the file transfer module.

[0209] 2. Data preprocessing process

[0210] Step 1: Create a file transfer message queue on the file transfer module.

[0211] Step 2: The file parsing module subscribes to the message queue of the file transfer module.

[0212] Step three: The editor publishes the page by operating the content management and publishing module. The page published by the content management and publishing module is uploaded to the content distribution network module source station through the file transfer module. While uploading, the change information such as file addition, deletion and modification is sent to the message queue.

[0213] Step 4: The file parsing module obtains the change information of the source site file in the message queue.

[0214] Step 5: The file parsing module reads the source file from the source station and parses the file according to the file information in the message queue. The main fields of the file parsing are title, body, release time, file path, file format, and audio and video GUID.

[0215] Step six, if there is a player in the file, you can extract the unique GUID of the audio and video based on the playback information, and then call the AI ​​middle station through the GUID to obtain the file text description of audio and video (including text such as character information and location).

[0216] Step seven, establish a full-text index library for the document parsing content for use by editors and queries.

[0217] Among them, the text of the HTML file is extracted using the method of the largest text block, the specific steps are:

[0218] First, parse the tags in the HTML file (such as body, div, p, etc.) into a DOM tree, record the total number of tags NodeCount (the minimum value is 1) and the position information NodePos (the minimum value is 1) of each tag.

[0219] Second, count the number of br tags (brCount) and p tags (pCount) in each node and save them in the metadata of each node. The calculation is brCount + 2 Sort by pCount and take the first N (currently N=12) nodes.

[0220] Third, count the number of links dLinkPower (minimum value is 1), the number of Chinese punctuation marks nCDotNum (minimum value is 1), the number of words WN (minimum value is 1), and the number of words with links LWL in the first N nodes.

[0221] Fourth, calculate the second weight value of the node:

[0222] W2 = (WN-LWL) × dl × f

[0223] Wherein, W2 is the second weight value of each node, WN is the total number of words of each node, LWL is the number of linked words of each node, dl is the location information of each node, and f is the weight factor of each node.

[0224] According to the second weight values ​​of all the nodes in the top N, sort them again from large to small.

[0225] Among them, the position information of each node is obtained by the following expression:

[0226]

[0227] Among them, dl is the location information of each node, NodeCount and NodePos are the total number of tags in the HTML file and the location information of each tag, respectively.

[0228] The weight factor of each node is obtained by the following expression:

[0229]

[0230] Among them, f is the weight factor of each node, dLinkPower is the number of links of each node, and nCDotNum is the number of Chinese punctuation marks of each node.

[0231] Fifth, sort by the second weight from high to low, and take the first node as the largest body text block.

[0232] Sixth, extract the title of the article from the position of the text block node, and use regular expressions to extract the article release time. If the extraction is successful, the text in the node is the article body. If not, take the second node with the second weight and repeat this step.

[0233] Seventh, if the main content of the article cannot be extracted in step six or the length of the extracted main content is less than 100 words, then directly remove the html tags as the content of the article.

[0234] It should be understood that, although the various steps in the flow chart are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear description in this article, the execution of these steps is not strictly limited in order, and these steps can be executed in other orders. Moreover, at least a portion of the steps in the figure may include multiple sub-steps or multiple stages, and these sub-steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these sub-steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a portion of the sub-steps or stages of other steps.

[0235] See also Figure 6 An embodiment of the present application provides a device 600 for quickly searching and deleting a published file, comprising:

[0236] The file transmission module 610 is used to receive a target file, send the received target file to the content distribution network module, send information about the target file to the message server, and delete the target file to be deleted in the content distribution network module;

[0237] The message server 620 is used to provide information of the target file;

[0238] The file parsing module 630 is used to obtain information of a target file on a message server, obtain the target file, parse the target file, and push the parsed file title, time and text information to the indexing module;

[0239] The index module 640 is used to search for files that meet preset conditions according to the file title, time and text information, and send them to the file transmission module as target files to be deleted.

[0240] For the specific definition of the above-mentioned device 600, please refer to the definition of the method for quickly finding and deleting published files in the above text, which will not be repeated here. Each module in the above-mentioned device 600 can be implemented in whole or in part by software, hardware and a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory in the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0241] In one embodiment, a computer device is provided, the internal structure diagram of which can be as follows: Figure 7As shown. The computer device includes a processor, a memory, a network interface and a database connected by a module bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operation module, a computer program and a database. The internal memory provides an environment for the operation of the operation module and the computer program in the non-volatile storage medium. The database of the computer device is used to store data. The network interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a method for quickly finding and deleting published files as described above is implemented. It includes: a memory and a processor, the memory stores a computer program, and when the processor executes the computer program, any step in the method for quickly finding and deleting published files as described above is implemented.

[0242] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, any step in the method for quickly finding and deleting published files can be implemented.

[0243] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, modules, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present application may adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program codes.

[0244] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (modules), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0245] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0246] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0247] Although the preferred embodiments of the present application have been described, those skilled in the art may make other changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications falling within the scope of the present application.

[0248] Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is also intended to include these modifications and variations.

Claims

1. A method for quickly finding and deleting published files, characterized in that: include: Receive the target file by using the file transfer module, send the received target file to the content distribution network module, and send the information of the target file to the message server; Use the file parsing module to obtain the information of the target file on the message server, obtain the target file, parse the target file, and push the parsed file title, time and text information to the index module; The index module is used to search for files that meet the preset conditions according to the file title, time and text information, and the files are sent to the file transfer module as target files to be deleted; Utilize the file transfer module to delete the target file to be deleted in the content distribution network module. The index module is used to search for files that meet the preset conditions based on the file title, time and text information as target files to be deleted, including: Input the file title, time and text information of the specified file into the pre-trained file evaluation model; When the evaluation result output by the model indicates that the specified file is a target file to be deleted, the specified file is used as the target file to be deleted. Among them, the file evaluation model is a multimodal big model. Microservices are established through the multimodal big model, and a file evaluation model based on the multimodal big model is formed. The file evaluation model architecture based on the multimodal big model includes hardware, software, database and functional modules. The functional modules include system data conversion module, content review algorithm encapsulation module, system output module, unstructured data processing module, multimodal data integration module, content analysis module, content anomaly quantification module and intelligent control module. The unstructured data processing module is used to perform semantic analysis on news graphics, audio, and video unstructured data using a multimodal large model, generate a semantic association graph, extract the core context and information relevance in text and images through in-depth analysis, and construct a preliminary review data set; The multimodal data integration module is used to integrate the graphic data, audio data and video data in a unified context, convert them into a graphic-text association vector space, and transmit them to the content analysis module for in-depth analysis of content context and emotion; The content analysis module is used to preliminarily determine the matching degree between the text and the image content based on the semantic association graph generated by the unstructured data processing module.

2. The method according to claim 1, characterized in that: The file parsing module is used to obtain the target file information on the message server, obtain the target file, parse the target file, and push the parsed file title, time and text information to the index module, including: Get the file type of the target file; According to the file type of the target file, determine whether the target file needs to be preprocessed; When the target file needs to be preprocessed, the target file is preprocessed, and then the preprocessed target file is parsed; Without preprocessing the target file, the target file is directly parsed; Push the parsed file title, time and text information to the index module.

3. The method according to claim 2, characterized in that According to the file type of the target file, determine whether the target file needs to be preprocessed, including: When the file type of the target file is a text file, there is no need to preprocess the target file; When the file type of the target file is a JSON file or an HTML file, the target file needs to be preprocessed.

4. The method according to claim 3, characterized in that When the target file needs to be preprocessed, the target file is preprocessed, including: When the file type of the target file is a JSON file, delete the JSON tag in the JSON file; When the file type of the target file is an HTML file, and the HTML file contains at least one of audio and video, obtain the globally unique identifiers of the audio and video, obtain the identification text of the audio and video in the HTML file through the globally unique identifiers of the audio and video, and then use the identification text as the HTML file to extract the largest body text block of the HTML file; when the HTML file does not contain at least one of audio and video, extract the largest body text block of the HTML file.

5. The method according to claim 4, characterized in that Extract the largest body text block of an HTML file, consisting of: Parse the tags in the HTML file into a DOM tree and obtain the total number of tags in the HTML file and the location information of each tag; According to the number of line break tags and paragraph tags in each node of the HTML file, all nodes are sorted to get the top N nodes; For each of the top N nodes, perform secondary sorting based on the number of links, the number of Chinese punctuation marks, the total number of words and the number of words with links, the total number of tags in the HTML file, and the position information of each tag, and take the first node as the largest text block; Move forward from the node position of the largest text block, extract the title of the HTML file, extract the release time of the HTML file through a regular expression, and if the extraction is successful, use the text of the first node as the text of the HTML file; if the extraction is unsuccessful, use the second node in the secondary sorting as the largest text block, and extract the title and release time of the HTML file until the extraction is successful; When the first node is taken as the largest body text block, if the body of the first node cannot be extracted or the number of extracted body words is less than a preset threshold, the html tag in the HTML file is directly deleted, and the HTML file with the html tag deleted is taken as the body of the HTML file.

6. The method according to claim 5, characterized in that According to the number of line break tags and paragraph tags in each node on the DOM tree, all nodes are sorted to get the top N nodes, including: The first weight value of each node is determined by the following expression: W1 = brCount + 2 pCount; Where W1 is the first weight value of each node, brCount is the number of line break tags of each node, and pCount is the number of paragraph tags of each node. All nodes are sorted from largest to smallest according to their first weight values ​​to obtain the top N nodes.

7. The method according to claim 5, characterized in that For each of the top N nodes, perform secondary sorting based on the number of links, the number of Chinese punctuation marks, the total number of words and the number of words with links, the total number of tags in the HTML file, and the position information of each tag, including: The second weight value of each node is determined by the following expression: W2 = (WN-LWL) × dl × f; Wherein, W2 is the second weight value of each node, WN is the total number of words of each node, LWL is the number of linked words of each node, dl is the location information of each node, and f is the weight factor of each node. According to the second weight values ​​of all the nodes in the top N, sort them again from large to small. Among them, the position information of each node is obtained by the following expression: ； Among them, dl is the location information of each node, NodeCount and NodePos are the total number of tags in the HTML file and the location information of each tag, respectively. The weight factor of each node is obtained by the following expression: ； Among them, f is the weight factor of each node, dLinkPower is the number of links of each node, and nCDotNum is the number of Chinese punctuation marks of each node.

8. A device for quickly searching and deleting published files, characterized in that: include: A file transmission module, used for receiving a target file, sending the received target file to the content distribution network module, sending information of the target file to the message server, and also for deleting the target file to be deleted in the content distribution network module; A message server, used to provide information about target files; The file parsing module is used to obtain the information of the target file on the message server, obtain the target file, parse the target file, and push the parsed file title, time and text information to the indexing module; The index module is used to search for files that meet the preset conditions based on the file title, time and text information, and send them to the file transfer module as the target files to be deleted. The index module is used to search for files that meet the preset conditions based on the file title, time and text information as target files to be deleted, including: Input the file title, time and text information of the specified file into the pre-trained file evaluation model; When the evaluation result output by the model indicates that the specified file is a target file to be deleted, the specified file is used as the target file to be deleted. Among them, the file evaluation model is a multimodal big model. Microservices are established through the multimodal big model, and a file evaluation model based on the multimodal big model is formed. The file evaluation model architecture based on the multimodal big model includes hardware, software, database and functional modules. The functional modules include system data conversion module, content review algorithm encapsulation module, system output module, unstructured data processing module, multimodal data integration module, content analysis module, content anomaly quantification module and intelligent control module. The unstructured data processing module is used to perform semantic analysis on news graphics, audio, and video unstructured data using a multimodal large model, generate a semantic association graph, extract the core context and information relevance in text and images through in-depth analysis, and construct a preliminary review data set; The multimodal data integration module is used to integrate the graphic data, audio data and video data in a unified context, convert them into a graphic-text association vector space, and transmit them to the content analysis module for in-depth analysis of content context and emotion; The content analysis module is used to preliminarily determine the matching degree between the text and the image content based on the semantic association graph generated by the unstructured data processing module.

9. A computer device comprising: A memory and a processor, wherein the memory stores a computer program, and wherein the processor implements the steps of the method for quickly finding and deleting published files as described in any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method for quickly finding and deleting published files as described in any one of claims 1 to 7 are implemented.

Images