A batch information retrieval and structured output method and system

Abstract

The present invention discloses a method and system for batch information retrieval and structured output, which belongs to the field of big data analysis and information security technology. It realizes batch information retrieval and structured output based on multi-source heterogeneous file parsing, and specifically includes: resource index construction and data processing: based on the data provided by the big data platform, through custom configuration, single resources and integrated resources are integrated into the search engine through configuration; and field mapping information is stored in the configuration database to realize whether the field is mapped for batch query; dynamic parsing engine and element extraction: through the file header feature code, the format type of the uploaded file is dynamically identified, and the key element information including ID number and mobile phone number is extracted using streaming parsing and hybrid classification model; distributed batch retrieval and query optimization; structured output and security compliance control. The present invention can significantly improve the efficiency and security of the fusion processing of multi-source data in scenarios such as public security investigation and financial anti-fraud.

Description

Technical Field

[0001] The present invention relates to the field of big data analysis and information security technology, and specifically to a batch information retrieval and structured output method and system. Background Art

[0002] With the surge in demand for multi-source data integration in public security scenarios, the industry faces the triple challenge of accurately parsing heterogeneous files, discovering cross-dimensional correlations, and generating compliant output. Current technology suffers from significant flaws: At the file parsing level, traditional systems rely on file extensions to identify formats (e.g., identifying Excel files as .xlsx), resulting in a misclassification rate of up to 7% for files whose content doesn't match the extension (e.g., CSV data disguised as .txt). Furthermore, optical character recognition (OCR) extraction technology for scanned documents lacks semantic segmentation capabilities, leading to frequent errors in the segmentation of key fields such as ID numbers. At the batch retrieval level, the existing distributed architecture utilizes a fixed thread pool to process heterogeneous tasks, without dynamically allocating computing resources based on data type. This results in searches of tens of thousands of data items taking over 180 seconds. Furthermore, the generated reports lack invisible digital watermarks, failing to meet the traceability and dissemination control requirements of regulations such as the GDPR. These issues lead to systemic deficiencies in existing solutions in terms of processing efficiency (response time > 3 minutes), parsing accuracy (field error rate > 5%), and compliance risk (data breach tracing failure rate 32%), severely hindering the value mining and security applications of multi-source data. Summary of the Invention

[0003] The technical task of the present invention is to address the above shortcomings and provide a batch information retrieval and structured output method and system, which can significantly improve the efficiency and security of the fusion processing of multi-source data in scenarios such as public security investigation and financial anti-fraud.

[0004] The technical solution adopted by the present invention to solve its technical problem is:

[0005] A batch information retrieval and structured output method is implemented based on multi-source heterogeneous file parsing, specifically including:

[0006] Resource index construction and data processing: Based on the data provided by the big data platform, individual resources and integrated resources are integrated into the search engine through custom configuration. Field mapping information is stored in the configuration database to implement mapping of fields for batch queries.

[0007] Dynamic parsing engine and feature extraction: Dynamically identifies the format type of uploaded files through file header signatures, and uses streaming parsing and hybrid classification models to extract key elements such as ID card numbers and mobile phone numbers;

[0008] Distributed batch retrieval and query optimization: Build a distributed retrieval task scheduling framework, dynamically allocate computing resources, and combine the improved PageRank algorithm to generate a user affinity ranking;

[0009] Structured output and security compliance control: Automatically synthesize structured Word reports based on the template engine and integrate digital watermarks.

[0010] This method can solve key technical bottlenecks in the parsing of multi-source heterogeneous files, such as high format misjudgment rate, low cross-modal data association efficiency, and insufficient security compliance. Through a dynamic file parsing engine, intelligent association analysis algorithm, and security control mechanism, it can achieve accurate parsing, second-level batch retrieval, and compliant structured output of massive heterogeneous data (such as Excel, PDF, CSV, etc.), thereby significantly improving the efficiency and security of multi-source data fusion processing in scenarios such as public security investigation and financial anti-fraud.

[0011] Furthermore, the resource index construction and data processing include the following steps:

[0012] Batch search field configuration: Based on business scenarios in the public security field, first preset the batch search type, including ID number, license plate number, mobile phone number, case number, etc.; then configure the resources, covering the data source, table name, Chinese name, primary key field, and timestamp field of the resource table; then configure the search engine information, including the search engine type, engine address, index name, number of instances, shards, replica information, and log writing method; finally, configure the mapping field, clarify the index field name, field Chinese description, word segmentation rules, weight, and select the corresponding batch search type. After completing the above configuration, the configuration content is stored in the configuration database, and the corresponding index is created in the search engine based on the configuration;

[0013] Resource index extraction: Based on specific query conditions (set as Query Condition), data records are read from the database (such as relational databases such as MySQL and PostgreSQL) through the corresponding database connection interface JDBC (assuming the read data record set is DataSet={Record1, Record2,…, Recordn}). Then, each data record is converted and mapped according to the data format requirements configured above. Finally, the converted and mapped data (TransformedDataSet) is written in batches or one by one to the specified index (Index) and document type (Type) of the search engine through the search engine client interface (such as the Java High Level REST Client of the search engine ES), thus completing the data extraction process from the database to the search engine, thereby realizing functions such as efficient data retrieval and analysis in the search engine.

[0014] Furthermore, each data record is converted and mapped according to the data format requirements of the above configuration.

[0015] Let the transformation function be TransformationFunction, and the result after mapping is:

[0016] TransformedDataSet={TransformationFunction(Record1),TransformationFunction(Record2),…,TransformationFunction(Recordn)}.

[0017] Furthermore, the dynamic parsing engine and element extraction specifically include:

[0018] File format identification: File formats uploaded in the public security sector include text files (such as .doc, .txt), image files (such as .jpg, .png), binary structured files (such as .xlsx), and PDF files. The parsing engine first uses the file extension, file header information, or specific identifiers to preliminarily determine the file format, including text files (such as .doc, .txt), image files (such as .jpg, .png), binary structured files (such as .xlsx), and PDF files. For example, the file header of a PDF file often contains identifiers such as "%PDF-," and image files have specific image format tags. The parsing engine uses these features to quickly classify the file.

[0019] Data extraction and conversion: Use corresponding extraction methods for files of different formats. After receiving the file using Spring Boot's MultipartFile interface, create a corresponding Workbook object based on the extension and obtain the Sheet worksheet. Traverse the rows and cells of the worksheet, extract data based on the cell type, and handle null values ​​and exceptions. After extraction, convert the data according to business requirements, such as processing date formats and calculating values. Finally, store the converted data in a structure such as List or Map for subsequent business processing.

[0020] Information parsing and extraction: In text content, ID card numbers, license plate numbers, mobile phone numbers, case numbers, etc. usually have specific encoding rules; for information such as ID card numbers and license plate numbers in images, optical character recognition (OCR) technology can be used to convert the text in the image into editable text, and then regular expressions can be used for matching and extraction.

[0021] Furthermore, for files of different formats, corresponding extraction methods are adopted;

[0022] For text files, directly read the text content and then, through operations such as character encoding conversion, unify it into a format convenient for processing; for image files, utilize image processing technology and integrate OCR services to extract element information such as ID numbers in the image files; for PDF files, use a PDF parsing library to extract text content and metadata; for excel files, use the Apache POI library to process the files.

[0023] Furthermore, for the distributed batch retrieval and query optimization, a large amount of data is dispersed and stored on multiple nodes, and each node is responsible for a part of the data; for example, according to a certain hash algorithm or data range division, the data is sharded to different nodes. For instance, the data is allocated to different nodes according to the hash value of the ID number, ensuring that the same or related data is stored in the same shard or adjacent nodes as much as possible for subsequent retrieval; when searching, a parallel retrieval method is adopted. When a batch retrieval request is initiated, the query is sent in parallel to multiple nodes storing the relevant data; multiple nodes perform search operations simultaneously, greatly accelerating the search speed. For example, when retrieving a batch of information such as ID numbers and license plate numbers, different nodes can simultaneously search the data they store instead of sequentially searching each node one by one; after each node completes the search, the results are returned to a coordination node or the client; the coordination node will perform operations including merging, sorting, etc. on these results to finally obtain a complete result set that meets the requirements.

[0024] Query optimization includes index optimization and query statement optimization.

[0025] The index optimization: First, reasonably design the index structure. According to the fields and query patterns that are frequently queried, create appropriate indexes; including, create separate indexes for fields such as ID numbers, license plate numbers, and contact information to improve query speed; for fields such as case numbers that may have specific format and range queries, design corresponding index structures to optimize the query; as the number of documents increases, the size of the index will also increase. To improve storage efficiency and retrieval speed, it is necessary to compress the inverted index. Adopt technologies such as differential coding to compress the index, reduce the storage space, and improve the read and write efficiency of the index between memory and disk. Assuming the original index size is S and the compression ratio is r (0 < r < 1), the size of the compressed index is ;

[0026] The query statement optimization: First, for fields with exact matches, such as ID numbers and license plate numbers, use precise term queries instead of match queries that may perform word segmentation, etc.; for example, when querying for an ID number of 123456789012345678, using a term query can directly locate the accurate record, while a match query may perform word segmentation on the ID number, resulting in a decrease in query efficiency.

[0027] Build efficient Boolean queries by reasonably combining clauses such as must (must be satisfied), should (should be satisfied), and mustNot (must not be satisfied) to construct Boolean queries and accurately filter data. For example, to query records that meet both a specific ID number and a case number range, you can use the must clause to combine these two conditions. Use filtering conditions to add filtering conditions to the query to reduce the number of documents that need to be scanned. For example, first filter out most irrelevant data through some fixed conditions such as area codes, and then perform more detailed queries. Assuming that the original query requires scanning N documents, and the filtering conditions can exclude M irrelevant documents, the number of documents that need to be scanned after optimization is Cache frequently used query results. When the same query is initiated again, the results are directly obtained from the cache without executing the query operation again. Let the cache hit rate be p (0≤p≤1). If the query execution time is T when there is no cache, then the average query time when there is a cache is , where t is the time to get the result from the cache, usually ( ).

[0028] Furthermore, the structured output and security compliance control, structured output includes batch search result export and personnel file information export;

[0029] The batch search result output: exporting the distributed search results in a structured manner;

[0030] Export of personnel file information: The exported personnel file includes basic information of the personnel, analysis of cohabitants, analysis of fellow travelers, intimacy ranking and other information;

[0031] Co-resident analysis: Based on a big data platform, potential co-residents are identified through address similarity calculation and association rule mining. Address data is first cleaned (e.g., removing special symbols and unifying administrative abbreviations), and key fields (e.g., community name and building number) are extracted for structured storage. Levenshtein distance is then used to quantify address differences, and a threshold is set to filter out highly similar records. The filtered similarity paths are then verified with multi-source data, for example, by combining auxiliary data such as utility billing records and property registration information to improve the accuracy of co-resident relationship judgments.

[0032] The described companion identification: extracts travel information with temporal and spatial overlap from traffic records (flights, high-speed trains, buses, etc.) or mobile phone signaling data, clusters trajectory points that appear at the same location or within a specified location within the same time period, and generates a list of companion events; then performs feature modeling from two aspects: time window and spatial density, defines a time tolerance (such as ±10 minutes) to offset device time errors, identifies dense trajectory areas based on the DBSCAN algorithm, and filters out accidental companions; counts multiple companion events, and assigns higher association weights to high-frequency companion combinations; frequency weight calculation: counts multiple companion events, and assigns higher association weights to high-frequency companion combinations, such as: companion intensity = number of companions × log (total travel duration).

[0033] The intimacy ranking: integrates various relationship data between people, including friend relationships in social networks, colleague relationships at work, kinship relationships in families, etc., removes invalid data and duplicate data, and converts the data into a format suitable for algorithm processing, such as constructing a graph structure, in which nodes represent people and edges represent relationships between people; improves on the basis of the traditional PageRank algorithm, considers more factors that affect the intimacy of people, for example, considers the strength of the relationship (such as frequent contact has higher relationship strength), the type of relationship, etc., assigns corresponding weights to the edges and nodes in the graph structure, applies the improved algorithm to the constructed graph structure, and updates the PageRank value of each node through iterative calculation; in the calculation process, according to the weight of the edge and the connection relationship between the nodes, the importance score of the node is transmitted and assigned; finally, according to the calculated PageRank value of each person node, the people are sorted to generate a person intimacy ranking; the higher the PageRank value of the person, the higher the position in the intimacy ranking, indicating the intimacy with other people, and the top 5 people in the intimacy ranking are output;

[0034] Through the above calculations and analysis, the basic information of the personnel file (name, gender, contact information, etc.) and related information are finally output: a list of cohabitants and fellow travelers and their frequency of occurrence, and the top five people with the highest intimacy (personnel name, ID number, contact information, intimacy, etc.);

[0035] Security and compliance control: Automatically add a watermark to the exported results. The watermark content is the name and ID number of the currently logged-in user.

[0036] The present invention also claims protection for a batch information retrieval and structured output system, comprising:

[0037] The resource index construction and data processing module is used to integrate individual resources and integrated resources into the search engine through custom configuration based on the data provided by the big data platform. It also stores field mapping information in the configuration database to implement mapping of fields for batch queries.

[0038] The dynamic parsing engine and feature extraction module is used to dynamically identify the format type of uploaded files through file header signatures, and uses streaming parsing and hybrid classification models to extract key element information such as ID card number and mobile phone number;

[0039] Distributed batch retrieval and query optimization module, used to build a distributed retrieval task scheduling framework, dynamically allocate computing resources, and generate person affinity rankings using an improved PageRank algorithm;

[0040] Structured output and security compliance control module, used to automatically synthesize structured Word reports based on the template engine and integrate digital watermarks;

[0041] The system realizes batch information retrieval and structured output through the above method.

[0042] The present invention also claims a batch information retrieval and structured output device comprising: at least one memory and at least one processor;

[0043] The at least one memory is configured to store a machine-readable program;

[0044] The at least one processor is configured to call the machine-readable program to implement the above method.

[0045] The present invention also claims protection for a computer-readable medium having computer instructions stored thereon, which are capable of implementing the above method when executed by a processor.

[0046] Compared with the prior art, the batch information retrieval and structured output method and system of the present invention have the following beneficial effects:

[0047] 1. This invention dynamically identifies uploaded file formats through a combination of file header signatures and extensions. Compared to traditional methods that rely solely on extensions to identify formats, it effectively reduces the misjudgment rate for abnormal files (such as CSV data disguised as .txt), achieving a file parsing error rate of less than 0.3%. Furthermore, the use of OCR technology with integrated semantic segmentation capabilities for scanned documents significantly reduces segmentation errors in key fields such as ID numbers, significantly improving the accuracy and consistency of multi-source heterogeneous file parsing and providing a reliable foundation for subsequent data processing and analysis.

[0048] 2. The present invention constructs a distributed retrieval task scheduling framework, dynamically allocates computing resources according to data types, and adopts a parallel retrieval method, so that the system can achieve a second-level response for 100,000-level data. Compared with the existing distributed architecture that uses a fixed thread pool to process heterogeneous tasks, resulting in a retrieval of 10,000-level data taking more than 180 seconds, the retrieval efficiency of the present invention has been greatly improved. In addition, through index optimization and query statement optimization, such as the reasonable design of index structure, the use of precise term queries, and the construction of efficient Boolean queries, the data retrieval speed is further accelerated, meeting the needs of public security investigations, corporate background checks and other scenarios for rapid retrieval of massive data.

[0049] 3. This invention integrates association analysis technology to mine potential relationships between multimodal data such as identity, communication, and trajectory. For example, in cohabitation analysis, address similarity calculation and association rule mining, combined with auxiliary data such as utility bill records, accurately identify potential cohabitants. For companion identification, temporal and spatial overlap information is extracted from traffic records or mobile phone signaling data, and through feature modeling and filtering, companions are accurately identified. Based on an improved PageRank algorithm, a variety of relationship data is integrated to generate a closeness ranking of individuals. These features build a comprehensive multidimensional association network, addressing the lack of cross-dimensional association analysis in traditional systems and providing strong support for deeply mining the value of data.

[0050] 4. While outputting structured data, this invention automatically adds a watermark to the exported results. The watermark contains the name and ID number of the currently logged-in user. This method meets the requirements of GDPR and other regulations for operation traceability and dissemination control, and to a certain extent prevents data leakage. Even if a data leakage incident occurs, the watermark information can be traced back to the source of the data outflow, effectively reducing the failure rate of data leakage tracing and improving data compliance and security. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] Figure 1 It is a flowchart of a method for batch information retrieval and structured output provided by one embodiment of the present invention. DETAILED DESCRIPTION

[0052] The present invention will be further described below with reference to specific embodiments.

[0053] The embodiment of the present invention provides a method for batch information retrieval and structured output, which implements batch information retrieval and structured output based on multi-source heterogeneous file parsing, specifically including:

[0054] 1. Resource index construction and data processing:

[0055] Based on the data provided by the big data platform, through custom configuration, individual resources and integrated resources are integrated into the search engine; and the field mapping information is stored in the configuration database to realize whether the field is mapped for batch query. The following steps are included:

[0056] (1) Batch search field configuration: Based on the business scenarios in the public security field, first preset the batch search type, including ID number, license plate number, mobile phone number, case number, etc.; then configure the resources, covering the data source, table name, Chinese name, primary key field, and timestamp field of the resource table; then configure the search engine information, including the search engine type, engine address, index name, number of instances, shards, replica information, and log writing method; finally, configure the mapping field, clarify the index field name, field Chinese description, word segmentation rules, weight, and select the corresponding batch search type. After completing the above configuration, store the configuration content in the configuration database, and create the corresponding index in the search engine based on the configuration.

[0057] (2) Resource index extraction: Based on specific query conditions (set as Query Condition), data records are read from the database (such as relational databases such as MySQL and PostgreSQL) through the corresponding database connection interface JDBC (assuming the read data record set is DataSet = {Record1, Record2, …, Recordn}). Then, each data record is transformed and mapped according to the data format requirements of the above configuration (assuming the transformation function is TransformationFunction, and the result after mapping is: TransformedDataSet = {TransformationFunction(Record1), TransformationFunction(Record2), …, TransformationFunction(Recordn)}). Finally, the transformed and mapped data (TransformedDataSet) is written in batches or one by one to the specified index (Index) and document type (Type) of the search engine through the search engine client interface (such as the Java High Level REST Client of the search engine ES), thus completing the data extraction process from the database to the search engine, so as to realize the functions of efficient data retrieval and analysis in the search engine.

[0058] 2. Dynamic parsing engine and feature extraction: Dynamically identify the format type of uploaded files through file header feature codes, and use streaming parsing and hybrid classification models to extract key element information including ID number and mobile phone number.

[0059] (1) File format identification: File formats uploaded in the public security field include text files (such as .doc, .txt, etc.), image files (such as .jpg, .png, etc.), binary structured files (such as .xlsx, etc.), and PDF files. The parsing engine first determines the file format based on the file extension, file header information, or specific identifiers. For example, the file header of a PDF file usually contains identifiers such as "%PDF-", and image files have specific image format tags. The parsing engine quickly classifies files based on these features.

[0060] (2) Data extraction and conversion: For files of different formats, corresponding extraction methods are used. For text files, the text content is directly read, and then it is unified into a format that is easy to process through operations such as character encoding conversion. For image files, image processing technology is used to integrate OCR services to extract element information such as ID card numbers from image files. For PDF files, the PDF parsing library is used to extract text content and metadata; for Excel files, the Apache POI library is used to process files. After receiving the file with the help of Spring Boot's MultipartFile interface, the corresponding Workbook object is created based on the extension and the Sheet worksheet is obtained. The rows and cells of the worksheet are traversed, and data is extracted according to the cell type, while handling null values ​​and exceptions. After extraction, the data is converted according to business needs, such as processing date format, calculating numerical values, etc. Finally, the converted data is stored in a structure such as List or Map for subsequent business processing.

[0061] (3) Information parsing and extraction: In text content, ID card numbers, license plate numbers, mobile phone numbers, case numbers, etc. usually have specific encoding rules. For information such as ID card numbers and license plate numbers in images, optical character recognition (OCR) technology can be used to convert the text in the image into editable text, and then regular expressions can be used to match and extract it. The following shows a set of regular expressions:

[0062] ID number:

[0063]

[0064] License plate number:

[0065]

[0066] phone number:

[0067]

[0068] Case Number:

[0069] .

[0070] 3. Distributed Batch Retrieval and Query Optimization: Build a distributed retrieval task scheduling framework to dynamically allocate computing resources, and combine with an improved PageRank algorithm to generate a ranking of personnel intimacy.

[0071] The principle of distributed batch retrieval is as follows:

[0072] A large amount of data is scattered and stored on multiple nodes, and each node is responsible for a part of the data. For example, according to a certain hash algorithm or data range division, the data is sharded to different nodes. Like distributing data to different nodes according to the hash value of the ID number, ensuring that the same or related data is stored in the same shard or adjacent nodes as much as possible, which is convenient for subsequent retrieval. When searching, a parallel retrieval method is adopted. When a batch retrieval request is initiated, the query will be sent to multiple nodes storing relevant data in parallel. Multiple nodes perform search operations simultaneously, greatly accelerating the search speed. For example, to retrieve a batch of information such as ID numbers and license plate numbers, different nodes can simultaneously search the data they store, rather than sequentially searching each node one by one. After each node completes the search, the results are returned to a coordination node or the client. The coordination node will perform operations such as merging, sorting, and collating on these results to finally obtain a complete result set that meets the requirements.

[0073] Query optimization includes index optimization and query statement optimization.

[0074] The index optimization: First, reasonably design the index structure. According to the fields and query patterns that are frequently queried, create appropriate indexes. For example, create separate indexes for fields such as ID numbers, license plate numbers, and contact information to improve query speed. For fields such as case numbers that may have specific format and range queries, design corresponding index structures to optimize the query. As the number of documents increases, the size of the index will also increase. To improve storage efficiency and retrieval speed, it is necessary to compress the inverted index. Adopt technologies such as differential coding to compress the index, reduce the storage space, and improve the read and write efficiency of the index between memory and disk. Assuming the original index size is S and the compression ratio is r (0 < r < 1), the size of the compressed index is .

[0075] The query statement optimization: First, for fields with exact matches, such as ID numbers and license plate numbers, use precise term queries instead of match queries that may perform word segmentation. For example, when querying an ID number 123456789012345678, using a term query can directly locate the accurate record, while a match query may perform word segmentation on the ID number, resulting in a decrease in query efficiency.

[0076] Build efficient Boolean queries by reasonably combining clauses including must (must be satisfied), should (should be satisfied), mustNot (must not be satisfied), etc. to build Boolean queries and accurately filter data; for example, to query records that meet both a specific ID number and a case number range, you can use the must clause to combine these two conditions. Use filter conditions to add filter conditions to the query to reduce the number of documents that need to be scanned. For example, first filter out most of the irrelevant data through some fixed conditions such as area codes, and then conduct more detailed queries. Assuming that the original query requires scanning N documents, and the filter conditions can exclude M irrelevant documents, the number of documents that need to be scanned after optimization is Cache frequently used query results. When the same query is initiated again, the results are directly obtained from the cache without executing the query again. Let the cache hit rate be p (0≤p≤1), and if the query execution time without cache is T, then the average query time with cache is , where t is the time to get the result from the cache, usually ( ).

[0077] 4. Structured output and security compliance control: Automatically synthesize structured Word reports based on the template engine and integrate digital watermarks.

[0078] Structured output includes exporting batch search results and personnel file information;

[0079] (1) Batch retrieval result output: Export the distributed search results from the previous step in a structured manner.

[0080] (2) Export of personnel file information: The exported personnel file information includes basic information of the personnel, analysis of co-residents, analysis of peers, intimacy ranking and other information.

[0081] Co-resident Analysis: Based on a big data platform, potential co-residents are identified through address similarity calculation and association rule mining. Address data is first cleaned (e.g., removing special symbols and standardizing administrative abbreviations), and key fields (e.g., community name and building number) are extracted for structured storage. Levenshtein distance is then used to quantify address differences, and a threshold is set to filter out highly similar records. The selected similarity paths are then validated with multi-source data, for example, by combining auxiliary data such as utility bills and property registration information, to improve the accuracy of co-residential relationship determination.

[0082] Companion identification: Extracting temporally and spatially overlapping travel information from traffic records (flights, high-speed rail, buses, etc.) or mobile phone signaling data. Clustering trajectory points that appear at the same or nearby locations within the same time period generates a list of co-travel events. Feature modeling is then performed based on both time windows and spatial density. A time tolerance (e.g., ±10 minutes) is defined to compensate for device time errors. The DBSCAN algorithm is used to identify dense trajectory areas and filter out accidental co-travel. Multiple co-travel events are counted, assigning higher association weights to frequent co-travel combinations. Frequency weight calculation: Multiple co-travel events are counted, assigning higher association weights to frequent co-travel combinations. For example, co-travel intensity = number of co-travels × log(total travel duration).

[0083] Affinity ranking: This algorithm integrates various relationship data between people, including friendships in social networks, colleagues at work, and family relationships. It removes invalid and duplicate data and converts the data into a format suitable for algorithmic processing, such as constructing a graph structure where nodes represent people and edges represent relationships between them. This algorithm improves upon the traditional PageRank algorithm by considering more factors influencing person affinities. For example, it considers not only the existence of a relationship but also the strength of the relationship (setting word affinities to 1 and multiplying them by the number of times a relationship has been established. For example, a single train journey has a 1 affinity of 2, while two train journeys have a 2 affinity). It also considers the type of relationship (setting different affinities based on the type of relationship, such as 2 for households in the same household, 2 for marriages, 1 for travelers in the same car, and 1 for violations committed on the same train). Weights are assigned to edges and nodes in the graph. The improved algorithm is applied to the constructed graph, and the PageRank value of each node is updated through iterative calculations. During the calculation process, node importance scores are transferred and assigned based on edge weights and the connections between nodes. Finally, the people are ranked based on the calculated PageRank values ​​of each person node to generate a person affinities ranking. The higher the PageRank value of a person, the higher their position in the intimacy ranking, indicating their closeness to other people. The top 5 people in the intimacy ranking are output.

[0084] Through the above calculations and analysis, the basic information of the personnel file (basic information such as name, gender, and contact information), related information: a list of co-residents and fellow travelers and their frequency of appearance, and the top five people with the highest intimacy (personnel name, ID number, contact information, intimacy, etc.) are finally output.

[0085] (3) Security and compliance control: Automatically add a watermark to the exported results. The watermark content is the name and ID number of the currently logged-in user.

[0086] This method integrates a dynamic file parsing engine, a distributed resource scheduling algorithm, and association analysis technology. It is particularly suitable for processing data sources in mixed formats such as Excel spreadsheets, scanned documents, and spatiotemporal trajectories. It can achieve accurate extraction of key elements such as ID card numbers and mobile phone numbers, construction of multi-dimensional association networks, and automatic generation of compliance reports. It solves the technical problems of poor consistency in multi-format file parsing, low batch retrieval efficiency, and lack of cross-dimensional association analysis in traditional systems. It can achieve a response of 100,000 data points in seconds, with a file parsing error rate of less than 0.3%. It can be widely used in public security investigations, corporate background checks, and other fields, significantly improving the depth of data utilization and compliance security.

[0087] An embodiment of the present invention further provides a batch information retrieval and structured output system, which implements batch information retrieval and structured output through the batch information retrieval and structured output method described in the above embodiment.

[0088] The system includes:

[0089] 1. Resource index construction and data processing module, which is used to integrate individual resources and integrated resources into the search engine through custom configuration based on the data provided by the big data platform; and store field mapping information in the configuration database to realize whether the field is mapped for batch query. It includes:

[0090] (1) Batch search field configuration: Based on the business scenarios in the public security field, first preset the batch search type, including ID number, license plate number, mobile phone number, case number, etc.; then configure the resources, covering the data source, table name, Chinese name, primary key field, and timestamp field of the resource table; then configure the search engine information, including the search engine type, engine address, index name, number of instances, shards, replica information, and log writing method; finally, configure the mapping field, clarify the index field name, field Chinese description, word segmentation rules, weight, and select the corresponding batch search type. After completing the above configuration, store the configuration content in the configuration database, and create the corresponding index in the search engine based on the configuration.

[0091] (2) Resource index extraction: Data records are read from a database (such as a relational database such as MySQL or PostgreSQL) through the corresponding database connection interface JDBC according to specific query conditions (set as Query Condition). Then, each data record is converted and mapped according to the data format requirements of the above configuration (set the transformation function as TransformationFunction, and the result after mapping is: TransformedDataSet={TransformationFunction(Record1),TransformationFunction(Record2),…,TransformationFunction(Recordn)}). Finally, the converted and mapped data (TransformedDataSet) is written in batches or one by one to the specified index (Index) and document type (Type) of the search engine through the search engine client interface (such as the Java High Level REST Client of the search engine ES), thereby completing the data extraction process from the database to the search engine, so as to realize the functions of efficient data retrieval and analysis in the search engine.

[0092] 2. Dynamic parsing engine and feature extraction module, used to dynamically identify the format type of uploaded files through file header feature codes, and use streaming parsing and hybrid classification models to extract key element information including ID card number and mobile phone number. Including:

[0093] (1) File format identification: File formats uploaded in the public security field include text files (such as .doc, .txt, etc.), image files (such as .jpg, .png, etc.), binary structured files (such as .xlsx, etc.), and PDF files. The parsing engine first determines the file format based on the file extension, file header information, or specific identifiers. For example, the file header of a PDF file usually contains identifiers such as "%PDF-", and image files have specific image format tags. The parsing engine quickly classifies files based on these features.

[0094] (2) Data extraction and conversion: For files of different formats, corresponding extraction methods are used. For text files, the text content is directly read, and then it is unified into a format that is easy to process through operations such as character encoding conversion. For image files, image processing technology is used to integrate OCR services to extract element information such as ID card numbers from image files. For PDF files, the PDF parsing library is used to extract text content and metadata; for Excel files, the Apache POI library is used to process files. After receiving the file with the help of Spring Boot's MultipartFile interface, the corresponding Workbook object is created based on the extension and the Sheet worksheet is obtained. The rows and cells of the worksheet are traversed, and data is extracted according to the cell type, while handling null values ​​and exceptions. After extraction, the data is converted according to business needs, such as processing date format, calculating numerical values, etc. Finally, the converted data is stored in a structure such as List or Map for subsequent business processing.

[0095] (3) Information parsing and extraction: In text content, ID card numbers, license plate numbers, mobile phone numbers, case numbers, etc. usually have specific encoding rules. For information such as ID card numbers and license plate numbers in images, optical character recognition (OCR) technology can be used to convert the text in the image into editable text, and then regular expressions can be used to match and extract it.

[0096] 3. Distributed batch retrieval and query optimization module, used to build a distributed retrieval task scheduling framework, dynamically allocate computing resources, and generate person intimacy rankings based on the improved PageRank algorithm.

[0097] The principle of distributed batch retrieval is as follows:

[0098] Large amounts of data are stored in a distributed manner across multiple nodes, with each node responsible for a portion of the data. For example, data can be sharded to different nodes based on a hash algorithm or data range, such as by assigning data to different nodes based on the hash value of an ID number. This ensures that identical or related data is stored in the same shard or on adjacent nodes whenever possible to facilitate subsequent retrieval. Parallel retrieval is used during searches. When a batch search request is initiated, the query is sent in parallel to multiple nodes storing related data. Having multiple nodes perform searches simultaneously greatly speeds up the search. For example, to retrieve a batch of information such as ID numbers and license plate numbers, different nodes can search their respective stored data simultaneously, rather than searching each node sequentially. After each node completes the search, it returns the results to a coordinating node or client. The coordinating node then performs operations such as merging, sorting, and sorting these results to ultimately obtain a complete, qualified result set.

[0099] Query optimization includes index optimization and query statement optimization.

[0100] The index optimization: First, rationally design the index structure. According to the frequently queried fields and query patterns, create appropriate indexes. For example, create separate indexes for fields such as ID card numbers, license plate numbers, contact information, etc. to improve query speed. For fields like case numbers that may have specific format and range queries, design corresponding index structures to optimize the queries. As the number of documents increases, the size of the index will also grow. To improve storage efficiency and retrieval speed, it is necessary to compress the inverted index. Adopt technologies such as differential coding to compress the index, reduce storage space, and improve the read and write efficiency of the index between memory and disk. Assuming the original index size is S and the compression ratio is r (0 < r < 1), the size of the compressed index is .

[0101] The optimization of query statements: First, for fields with exact matches, such as ID card numbers, license plate numbers, etc., use precise term queries instead of match queries that may perform word segmentation. For example, when querying for an ID card number of 123456789012345678, using a term query can directly locate the accurate record, while a match query may perform word segmentation on the ID card number, resulting in reduced query efficiency.

[0102] Construct an efficient boolean query. Build a boolean query by reasonably combining clauses including must (must be satisfied), should (should be satisfied), mustNot (must not be satisfied), etc. to accurately filter data. For example, when querying for records that simultaneously meet a specific ID card number and a case number range, the must clause can be used to combine these two conditions. Utilize filtering conditions and add filtering conditions in the query to reduce the number of documents to be scanned. For example, first filter out most irrelevant data through some fixed conditions such as area codes, and then conduct a more detailed query. Assuming the original query needs to scan N documents, and M irrelevant documents can be excluded through the filtering conditions, then the number of documents to be scanned after optimization is ; Cache frequently used query results. When the same query is initiated again, directly obtain the results from the cache without having to execute the query operation again. Let the cache hit rate be p (0 ≤ p ≤ 1). If the query execution time without caching is T, then the average query time with caching is , where t is the time to obtain the results from the cache, usually ( ).

[0103] 4. The structured output and security compliance control module is used to automatically synthesize a structured Word report based on a template engine and integrate digital watermarks.

[0104] The structured output includes batch retrieval result export and export of personnel file information;

[0105] Batch retrieval result output: Export the distributed search results from the previous step in a structured manner.

[0106] The export of personnel file information: The exported content of personnel files includes basic information of personnel, analysis of co-residents, analysis of peers, intimacy ranking and other information.

[0107] Co-resident Analysis: Based on a big data platform, potential co-residents are identified through address similarity calculation and association rule mining. Address data is first cleaned (e.g., removing special symbols and standardizing administrative abbreviations), and key fields (e.g., community name and building number) are extracted for structured storage. Levenshtein distance is then used to quantify address differences, and a threshold is set to filter out highly similar records. The selected similarity paths are then validated with multi-source data, for example, by combining auxiliary data such as utility bills and property registration information, to improve the accuracy of co-residential relationship determination.

[0108] Companion identification: Extracting temporally and spatially overlapping travel information from traffic records (flights, high-speed rail, buses, etc.) or mobile phone signaling data. Clustering trajectory points that appear at the same or nearby locations within the same time period generates a list of co-travel events. Feature modeling is then performed based on both time windows and spatial density. A time tolerance (e.g., ±10 minutes) is defined to compensate for device time errors. The DBSCAN algorithm is used to identify dense trajectory areas and filter out accidental co-travel. Multiple co-travel events are counted, assigning higher association weights to frequent co-travel combinations. Frequency weight calculation: Multiple co-travel events are counted, assigning higher association weights to frequent co-travel combinations. For example, co-travel intensity = number of co-travels × log(total travel duration).

[0109] Affinity ranking: This algorithm integrates various relationship data between people, including friendships in social networks, colleagues at work, and family relationships. It removes invalid and duplicate data and converts the data into a format suitable for algorithmic processing, such as constructing a graph structure where nodes represent people and edges represent relationships between them. This algorithm improves upon the traditional PageRank algorithm by considering more factors influencing person affinities. For example, it considers not only the existence of a relationship but also the strength of the relationship (setting word affinities to 1 and multiplying them by the number of times a relationship has been established. For example, a single train journey has a 1 affinity of 2, while two train journeys have a 2 affinity). It also considers the type of relationship (setting different affinities based on the type of relationship, such as 2 for households in the same household, 2 for marriages, 1 for travelers in the same car, and 1 for violations committed on the same train). Weights are assigned to edges and nodes in the graph. The improved algorithm is applied to the constructed graph, and the PageRank value of each node is updated through iterative calculations. During the calculation process, node importance scores are transferred and assigned based on edge weights and the connections between nodes. Finally, the people are ranked based on the calculated PageRank values ​​of each person node to generate a person affinities ranking. The higher the PageRank value of a person, the higher they are in the intimacy ranking, indicating their closeness to other people. The top 5 people in the intimacy ranking are output.

[0110] Through the above calculations and analysis, the basic information of the personnel file (basic information such as name, gender, and contact information), related information: a list of co-residents and fellow travelers and their frequency of appearance, and the top five people with the highest intimacy (personnel name, ID number, contact information, intimacy, etc.) are finally output.

[0111] Security and compliance control: Automatically add a watermark to the exported results. The watermark content is the name and ID number of the currently logged-in user.

[0112] An embodiment of the present invention further provides a batch information retrieval and structured output device, comprising: at least one memory and at least one processor;

[0113] The at least one memory is configured to store a machine-readable program;

[0114] The at least one processor is used to call the machine-readable program to implement the batch information retrieval and structured output method described in the above embodiment.

[0115] Embodiments of the present invention further provide a computer-readable medium storing computer instructions that, when executed by a processor, implement the batch information retrieval and structured output methods described in the above embodiments. Specifically, a system or device may be provided that includes a storage medium storing software program code that implements the functions of any of the above embodiments, and that enables a computer (or CPU or MPU) of the system or device to read and execute the program code stored in the storage medium.

[0116] In this case, the program code itself read from the storage medium can realize the function of any one of the above-mentioned embodiments, and thus the program code and the storage medium storing the program code constitute part of the present invention.

[0117] Examples of storage media for providing program code include floppy disks, hard disks, magneto-optical disks, optical disks (e.g., CD-ROMs, CD-Rs, CD-RWs, DVD-ROMs, DVD-RAMs, DVD-RWs, and DVD+RWs), magnetic tapes, non-volatile memory cards, and ROMs. Alternatively, the program code may be downloaded from a server computer via a communications network.

[0118] In addition, it should be clear that the functions of any of the above embodiments can be achieved not only by executing the program code read by the computer, but also by enabling the operating system operating on the computer to complete part or all of the actual operations based on the instructions of the program code.

[0119] In addition, it can be understood that the program code read from the storage medium is written into the memory provided in the expansion board inserted into the computer or into the memory provided in the expansion unit connected to the computer, and then based on the instructions of the program code, the CPU installed on the expansion board or expansion unit is enabled to perform part or all of the actual operations, thereby realizing the functions of any of the above embodiments.

[0120] The present invention has been shown and described in detail above through the accompanying drawings and preferred embodiments. However, the present invention is not limited to these disclosed embodiments. Based on the above multiple embodiments, those skilled in the art can know that the code review methods in the above different embodiments can be combined to obtain more embodiments of the present invention, and these embodiments are also within the scope of protection of the present invention.

Claims

1. A method for batch information retrieval and structured output, characterized in that: Batch information retrieval and structured output are achieved based on multi-source heterogeneous file parsing, specifically including: Resource index construction and data processing: Based on the data provided by the big data platform, individual resources and integrated resources are integrated into the search engine through custom configuration. Field mapping information is stored in the configuration database to implement mapping of fields for batch queries. Dynamic parsing engine and feature extraction: Dynamically identifies the format type of uploaded files through file header signatures, and uses streaming parsing and hybrid classification models to extract key elements such as ID card numbers and mobile phone numbers; Distributed batch retrieval and query optimization: Build a distributed retrieval task scheduling framework, dynamically allocate computing resources, and combine the improved PageRank algorithm to generate a user affinity ranking; Structured output and security compliance control: Automatically synthesize structured Word reports based on the template engine and integrate digital watermarks; The distributed batch search and query optimization method stores large amounts of data in a dispersed manner across multiple nodes, with each node responsible for a portion of the data. A parallel search method is used during search. When a batch search request is initiated, the query is sent in parallel to multiple nodes storing related data. Multiple nodes perform search operations simultaneously. After each node completes the search, the results are returned to a coordinating node or client. The coordinating node performs operations such as merging, sorting, and sorting the results to ultimately obtain a complete and satisfactory result set. Query optimization includes index optimization and query statement optimization. The index optimization: create indexes according to the frequently queried fields and query patterns, including: creating separate indexes for ID card numbers, license plate numbers, and contact information fields; designing corresponding index structures for fields with specific format and range queries; compressing inverted indexes. Let the original index size be S and the compression ratio be r, where 0 < r < 1. Then the size of the compressed index is ; The query statement optimization: First, for the fields that are exactly matched, use precise term query instead of match query that performs word segmentation; Build efficient Boolean queries by combining clauses including must, should, and mustNot to accurately filter data; add filtering conditions to queries to reduce the number of documents that need to be scanned; and cache frequently used query results.

2. A method for batch information retrieval and structured output according to claim 1, characterized in that: The resource index construction and data processing include the following steps: Batch search field configuration: Based on business scenarios in the public security field, first preset the batch search type, including ID number, license plate number, mobile phone number, and case number; then configure the resources, covering the data source, table name, Chinese name, primary key field, and timestamp field of the resource table; then configure the search engine information, including the search engine type, engine address, index name, number of instances, shards, replica information, and log writing method; finally, configure the mapping field, clarify the index field name, field Chinese description, word segmentation rules, weight, and select the corresponding batch search type. After completing the above configuration, the configuration content is stored in the configuration database, and the corresponding index is created in the search engine based on the configuration; Resource index extraction: Read data records from the database based on specific query conditions through the corresponding database connection interface JDBC; then, convert and map each data record according to the configured data format requirements; finally, write the converted and mapped data in batches or one by one to the search engine's specified index and document type through the search engine's client interface, thus completing the data extraction process from the database to the search engine.

3. A method for batch information retrieval and structured output according to claim 2, characterized in that: Convert and map each data record according to the data format requirements configured above. Let the transformation function be TransformationFunction, and the result after mapping is: TransformedDataSet={TransformationFunction(Record1),TransformationFunction(Record2),…,TransformationFunction(Recordn)}.

4. A method for batch information retrieval and structured output according to claim 1, characterized in that: The dynamic parsing engine and feature extraction specifically include: File format identification: The parsing engine first determines the file format based on the file extension, file header information, or specific identifiers, including text files, image files, binary structured files, and PDF files. Data extraction and conversion: Use corresponding extraction methods for files of different formats; after receiving the file using Spring Boot's MultipartFile interface, create the corresponding Workbook object based on the extension and obtain the Sheet worksheet; traverse the rows and cells of the worksheet, extract data based on the cell type, and handle null values ​​and exceptions; after extraction, convert the data according to business requirements and finally store the converted data in a List or Map structure; Information parsing and extraction: Use optical character recognition technology to convert the text in the image into editable text, and then use regular expressions to match and extract it.

5. A method for batch information retrieval and structured output according to claim 4, characterized in that: The corresponding extraction method is adopted for files of different formats; For text files, the text content is directly read and then converted into a format that is easy to process through character encoding operations. For image files, image processing technology is used to integrate OCR services to extract element information from image files. For PDF files, use a PDF parsing library to extract text content and metadata; For excel files, Apache POI library is used to process the files.

6. A method for batch information retrieval and structured output according to claim 1, characterized in that: The structured output and security compliance control include exporting batch search results and exporting personnel file information; The batch search result output: exporting the distributed search results in a structured manner; Export of personnel file information: The exported personnel file includes basic information of the personnel, analysis of cohabitants, identification of fellow travelers, and intimacy ranking information; The co-resident analysis is based on the big data platform and identifies potential co-residents through address similarity calculation and association rule mining. First, the address data is cleaned and key fields are extracted for structured storage. Then, we use edit distance to quantify address differences and set a threshold to filter out highly similar records. We then perform multi-source data verification on the filtered similar shortcuts. The companion identification: extracting spatiotemporal overlapping travel information from traffic records or mobile phone signaling data, clustering trajectory points that appear at the same location or within a specified location within the same time period, and generating a list of companion events; Then, feature modeling is performed from the perspectives of time window and spatial density. Time tolerance is defined to offset device time errors. Dense trajectory areas are identified based on the DBSCAN algorithm to filter out accidental peers. Multiple peer events are counted, and high-frequency peer combinations are assigned high correlation weights. Intimacy ranking: Integrate various relationship data between people, including friend relationships in social networks, colleague relationships at work, and kinship relationships in the family, remove invalid and duplicate data, convert the data into a format suitable for algorithm processing, and construct a graph structure in which nodes represent people and edges represent relationships between people; assign corresponding weights to the edges and nodes in the graph structure, and update the PageRank value of each node through iterative calculation; during the calculation process, the importance score of the node is transmitted and assigned based on the weight of the edge and the connection relationship between the nodes; finally, according to the calculated PageRank value of each person node, the people are sorted to generate a person intimacy ranking; the higher the PageRank value, the higher the person in the intimacy ranking, indicating their intimacy with other people, and the top 5 people in the intimacy ranking are output; Through the above calculations and analysis, the basic information and related information of the personnel profile are finally output: a list of co-residents, co-travelers and their frequency of appearance, and the top five people with the highest intimacy. Security and compliance control: Automatically add a watermark to the exported results. The watermark content is the name and ID number of the currently logged-in user.

7. A batch information retrieval and structured output system, characterized in that: include: The resource index construction and data processing module is used to integrate individual resources and integrated resources into the search engine through custom configuration based on the data provided by the big data platform. It also stores field mapping information in the configuration database to implement mapping of fields for batch queries. The dynamic parsing engine and feature extraction module is used to dynamically identify the format type of uploaded files through file header signatures, and uses streaming parsing and hybrid classification models to extract key element information such as ID card number and mobile phone number; Distributed batch retrieval and query optimization module, used to build a distributed retrieval task scheduling framework, dynamically allocate computing resources, and generate person affinity rankings using an improved PageRank algorithm; Structured output and security compliance control module, used to automatically synthesize structured Word reports based on the template engine and integrate digital watermarks; The distributed batch search and query optimization method stores large amounts of data in a dispersed manner across multiple nodes, with each node responsible for a portion of the data. A parallel search method is used during search. When a batch search request is initiated, the query is sent in parallel to multiple nodes storing related data. Multiple nodes perform search operations simultaneously. After each node completes the search, the results are returned to a coordinating node or client. The coordinating node performs operations such as merging, sorting, and sorting the results to ultimately obtain a complete and satisfactory result set. Query optimization includes index optimization and query statement optimization. The index optimization: Create indexes according to the frequently queried fields and query patterns, including: creating separate indexes for the ID card number, license plate number, and contact information fields; designing corresponding index structures for fields with specific format and range queries; compressing the inverted index. Let the original index size be S and the compression ratio be r, where 0 < r < 1. Then the compressed index size is ; The query statement optimization: First, for the fields that are exactly matched, use precise term query instead of match query that performs word segmentation; Build efficient Boolean queries by combining clauses including must, should, and mustNot to accurately filter data; add filtering conditions to queries to reduce the number of documents that need to be scanned; and cache frequently used query results.

8. A batch information retrieval and structured output device, characterized in that: include: at least one memory and at least one processor; The at least one memory is configured to store a machine-readable program; The at least one processor is configured to call the machine-readable program to implement the method according to any one of claims 1 to 6.

9. A computer-readable medium, characterized in that The computer-readable medium stores computer instructions, which, when executed by a processor, can implement the method according to any one of claims 1 to 6.

Images