Data processing method and device for fair competition review and medium

Abstract

The invention provides a data processing method and device for fair competition review and a medium, and relates to the technical field of fair competition review. The method comprises the following steps: performing protocol analysis and format conversion on automatically captured multi-source heterogeneous data by adopting a dynamic adapter library to obtain first data; performing data directional acquisition on the first data according to a preset dynamic acquisition strategy to obtain second data, the preset dynamic acquisition strategy being constructed according to the fair competition word bank; and determining a credibility weight corresponding to the second data, and constructing a corresponding fair competition knowledge graph according to the second data. According to the method provided by the invention, the efficiency, accuracy and comprehensiveness of data acquisition and fusion during fair competition examination are improved.

Description

Technical Field

[0001] This application relates to the field of fair competition review technology, and in particular to a data processing method, device and medium for fair competition review. Background Technology

[0002] Fair competition reviews require analysis of multi-dimensional data, including corporate market behavior (such as pricing strategies, market share, cooperation agreements, etc.), policies and regulations, industry reports, and social media sentiment. Therefore, obtaining comprehensive and accurate multi-dimensional data is particularly important.

[0003] In related technologies, the collection of multidimensional data for fair competition review typically relies on traditional data collection tools (such as general web crawlers) and manual data cleaning and fusion based on rules. However, this multidimensional data involves structured data (such as institutional databases and corporate annual reports), unstructured text (such as news and social media), images (such as corporate promotional materials), and streaming data (such as real-time transaction data). Due to the dispersed sources, diverse formats, and complex semantics of the data, traditional technologies struggle to achieve efficient and accurate data collection and fusion, leading to problems such as low review efficiency and insufficient credibility of the results.

[0004] Therefore, there is a need for a data processing solution that can improve the efficiency, accuracy, and comprehensiveness of data collection and fusion for fair competition review. Summary of the Invention

[0005] This application provides a data processing method, device, and medium for fair competition review, which can improve the efficiency, accuracy, and comprehensiveness of data collection and fusion during fair competition review.

[0006] In a first aspect, embodiments of this application provide a data processing method for fair competition review, including:

[0007] A dynamic adapter library is used to perform protocol parsing and format conversion on automatically captured multi-source heterogeneous data to obtain the first data.

[0008] The first data is collected in a targeted manner according to a preset dynamic collection strategy to obtain the second data. The preset dynamic collection strategy is constructed based on a fair competition lexicon.

[0009] Determine the credibility weight corresponding to the second data, and construct the corresponding fair competition knowledge graph based on the second data.

[0010] In one possible implementation, the step of using a dynamic adapter library to perform protocol parsing and format conversion on automatically crawled multi-source heterogeneous data to obtain first data includes:

[0011] Based on the protocol type of the automatically captured multi-source heterogeneous data, the corresponding protocol parsing module in the dynamic adapter library is determined, and the dynamic adapter library includes an extensible protocol parsing module;

[0012] The protocol parsing module is invoked to parse the communication protocol of the multi-source heterogeneous data to obtain the parsed data.

[0013] The parsed data is converted according to the format conversion module in the dynamic adapter library to obtain the first data in a unified format.

[0014] In one possible implementation, the dynamic acquisition strategy is constructed in the following way:

[0015] Construct a fair competition terminology database for different industries. The fair competition terminology database includes a keyword database, a legal basis database, and a case feature database. The keyword database includes core keywords and industry keywords. The legal basis database includes legal basis terms. The case feature database includes case feature terms.

[0016] For each industry, a corresponding dynamic data collection strategy is generated based on the industry's fair competition terminology database. The dynamic data collection strategy includes one or more of the following: monitoring data collection rules, price data collection rules, market segmentation data collection rules, and contract text data collection rules.

[0017] In one possible implementation, determining the credibility weight corresponding to the second data includes:

[0018] The structured data in the second data is subjected to structured processing to obtain processed structured data. The structured processing includes one or more of missing value repair, normalization, and deduplication.

[0019] The unstructured data in the second data is subjected to unstructured processing to obtain processed unstructured data. The unstructured processing includes one or more of entity error correction and contradiction detection.

[0020] Based on the processed structured data and the processed unstructured data, usable data is generated;

[0021] Based on the pre-trained credibility scoring model, a credibility weight is generated for each piece of available data.

[0022] In one possible implementation, the credibility scoring model is trained in the following ways:

[0023] Obtain sample data for fair competition review;

[0024] Based on the data dimensions of the sample data and the corresponding weight ratio of each data dimension, a credibility weight of the sample data is generated. The data dimensions include one or more of the following: data source authority, time freshness, multi-source consistency, content quality, and technical characteristics.

[0025] Based on the credibility weight of the sample data, generate sample labels for the sample data;

[0026] Based on the sample data and the sample labels of the sample data, the initial classification model is iteratively trained to obtain the credibility scoring model.

[0027] In one possible implementation, constructing the corresponding fair competition knowledge graph based on the second data includes:

[0028] Extract the initial entities corresponding to fair competition from the second data, and use graph embedding technology to map the initial entities to a unified vector space to perform entity alignment processing to obtain usable entities;

[0029] Determine the entity attributes corresponding to each available entity, whereby available entities include enterprises, policies, market behaviors, industries, regulations, and impacts;

[0030] Determine the relationships between the available entities and the relationship attributes of each relationship, including competitive relationships, regulatory relationships, compliance relationships, and violation relationships;

[0031] Based on each available entity, the entity attributes corresponding to each available entity, the relationships between each available entity, and the relationship attributes of each relationship, a corresponding knowledge graph of fair competition is constructed.

[0032] In one possible implementation, after constructing the corresponding fair competition knowledge graph based on the second data, the method further includes:

[0033] Collect real-time data of dynamic features and determine historical data of the dynamic features based on the fair competition knowledge graph;

[0034] A time window mechanism is used to associate and integrate the real-time data and the historical data to obtain integrated data;

[0035] The integrated data is input into a time series model to detect whether the dynamic features constitute abnormal behavior.

[0036] Based on the abnormal behavior detection results of the dynamic features, the data acquisition weights of the dynamic features are determined;

[0037] Data collection of multi-source heterogeneous data is carried out based on the data collection weights of dynamic features, static features, historical benchmark weights, and industry adjustment factors.

[0038] Secondly, embodiments of this application provide a computing device, including:

[0039] The data acquisition module is used to perform protocol parsing and format conversion on automatically captured multi-source heterogeneous data using a dynamic adapter library to obtain first data; and to perform targeted data acquisition on the first data according to a preset dynamic acquisition strategy to obtain second data, wherein the preset dynamic acquisition strategy is constructed based on a fair competition thesaurus.

[0040] The data fusion module is used to determine the credibility weight corresponding to the second data and to construct a corresponding fair competition knowledge graph based on the second data.

[0041] Thirdly, embodiments of this application provide another computing device, including:

[0042] The processor, and the memory that is in communication with the processor;

[0043] Memory is used to store instructions that the computer executes;

[0044] The processor is configured to execute computer execution instructions stored in memory, causing the processor to perform the first aspect and / or various possible implementations of the first aspect as described above.

[0045] Fourthly, embodiments of this application provide a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, are used to implement the first aspect and / or various possible implementations of the first aspect described above.

[0046] Fifthly, embodiments of this application provide a computer program product, including a computer program, which, when executed by a processor, is used to implement the first aspect and / or various possible implementations of the first aspect as described above.

[0047] This application provides a data processing method, device, and medium for fair competition review. It can automatically complete protocol parsing and format conversion of multi-source heterogeneous data using a dynamic adapter library, eliminating the need for manual intervention to adapt to different data sources. This significantly improves data aggregation efficiency, achieves standardized processing of scattered data, and provides unified and standardized foundational data for subsequent targeted data collection. A dynamic data collection strategy based on a fair competition thesaurus enables targeted data extraction, accurately filtering core information related to fair competition, avoiding irrelevant data redundancy, and improving the efficiency, accuracy, and comprehensiveness of data collection. Credibility weights can quantify the reliability of secondary data. A corresponding fair competition knowledge graph can be constructed based on the secondary data, enabling efficient, comprehensive, and accurate data association and fusion, providing comprehensive and accurate structured data for subsequent fair competition review. Attached Figure Description

[0048] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0049] Figure 1 This is a flowchart of a data processing method for fair competition review according to an embodiment of this application;

[0050] Figure 2 This is a schematic diagram of the structure of a dynamic adapter library according to an embodiment of this application;

[0051] Figure 3 This is a schematic diagram of the structure of a computing device according to an embodiment of this application;

[0052] Figure 4 This is a schematic diagram of the structure of a computing device according to another embodiment of this application.

[0053] The accompanying drawings have illustrated specific embodiments of this application, which will be described in more detail below. These drawings and descriptions are not intended to limit the scope of the concept in any way, but rather to illustrate the concept of this application to those skilled in the art through reference to specific embodiments. Detailed Implementation

[0054] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.

[0055] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a particular order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms “comprising” and “having,” and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0056] The collection, storage, use, processing, transmission, provision, and disclosure of financial data or user data involved in the technical solution of this application all comply with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0057] It should be noted that in the embodiments of this application, certain software, components, models and other existing solutions in the industry may be mentioned. These should be regarded as exemplary and are only intended to illustrate the feasibility of implementing the technical solution of this application. However, it does not mean that the applicant has used or necessarily used the solution.

[0058] It should also be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, use and processing of the relevant data must comply with relevant laws, regulations and standards, and corresponding operation entry points are provided for users to choose to authorize or refuse.

[0059] The data processing method, equipment, and media for fair competition review described in this application can be used in the field of fair competition review, or in any field other than fair competition review, such as the field of data processing. The application field of the data processing method, equipment, and media for fair competition review described in this application is not limited.

[0060] The data processing method, equipment, and media for fair competition review disclosed in this application can be applied to data collection scenarios conducted by market supervision departments before conducting fair competition reviews. The data processing method, equipment, and media for fair competition review disclosed in this application can be used for any processing of multi-source heterogeneous data related to fair competition review.

[0061] Fair competition reviews require analysis of multi-dimensional data, including corporate market behavior (such as pricing strategies, market share, cooperation agreements, etc.), policies and regulations, industry reports, and social media sentiment. Therefore, obtaining comprehensive and accurate multi-dimensional data is particularly important.

[0062] In related technologies, the multidimensional data involved in fair competition review is typically collected using traditional data collection tools (such as general web crawlers) and relies on manual rules for data cleaning and fusion. However, this multidimensional data includes structured data (such as institutional databases and corporate annual reports), unstructured text (such as news and social media), images (such as corporate promotional materials), and streaming data (such as real-time transaction data).

[0063] Due to the dispersed sources, diverse formats, and complex semantics of data, traditional technologies struggle to achieve efficient and accurate data collection and fusion, resulting in inefficient review processes and insufficient credibility of the results.

[0064] Based on the above-mentioned technical problems, the inventive concept of this application is to provide a data processing solution for fair competition review that can achieve high efficiency, accuracy and comprehensiveness in data collection and fusion.

[0065] This application provides a data processing method, device, and medium for fair competition review. It can automatically complete protocol parsing and format conversion of multi-source heterogeneous data using a dynamic adapter library, eliminating the need for manual intervention to adapt to different data sources. This significantly improves data aggregation efficiency, achieves standardized processing of scattered data, and provides unified and standardized foundational data for subsequent targeted data collection. A dynamic data collection strategy based on a fair competition thesaurus enables targeted data extraction, accurately filtering core information related to fair competition, avoiding irrelevant data redundancy, and improving the efficiency, accuracy, and comprehensiveness of data collection. Credibility weights can quantify the reliability of secondary data. A corresponding fair competition knowledge graph can be constructed based on the secondary data, enabling efficient, comprehensive, and accurate data association and fusion, providing comprehensive and accurate structured data for subsequent fair competition review.

[0066] The technical solution of this application and how the technical solution of this application solves the above-mentioned technical problems are described in detail below with specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments. The embodiments of this application will be described below with reference to the accompanying drawings.

[0067] Figure 1 This is a flowchart illustrating a data processing method for fair competition review according to an embodiment of this application. This embodiment describes the data processing method for fair competition review using a computing device as the executing entity. Figure 1 As shown, the data processing method for fair competition review may include the following steps:

[0068] S101: The dynamic adapter library is used to perform protocol parsing and format conversion on the automatically captured multi-source heterogeneous data to obtain the first data.

[0069] In this embodiment, the computing device can be a terminal device such as a computer or mobile phone, or a server device such as a physical server or cloud server, or an integrated terminal-server device; no restrictions are imposed here.

[0070] In this embodiment, the dynamic adapter library can build independent protocol parsing modules for different data sources (such as institutional databases, corporate websites, social media APIs, etc.). When data is crawled by connecting to the data source through the interface, the computing device can call the corresponding protocol parsing module to parse the communication protocol according to the protocol type in order to automatically obtain the data.

[0071] In this embodiment, the dynamic adapter library can be configured with an extensible protocol parsing module according to the protocol type to achieve automatic capture of various multi-source heterogeneous data without manual intervention to adapt to different data sources, thus greatly improving data aggregation efficiency.

[0072] In this embodiment, the dynamic adapter library may include a format conversion module, which can be used to convert various data formats (such as HTML, PDF, JSON, CSV, etc.) into a unified intermediate format, thereby achieving standardized processing of scattered data and providing unified and standardized basic data for subsequent targeted data collection.

[0073] For example, Figure 2 This is a schematic diagram of the structure of a dynamic adapter library according to an embodiment of this application, as shown below. Figure 2 As shown, the dynamic adapter library can include multiple protocol parsing modules and format conversion modules, and the protocol parsing modules can be dynamically adjusted and expanded.

[0074] S102: Collect the first data in a targeted manner according to the preset dynamic acquisition strategy to obtain the second data.

[0075] In this embodiment, the preset dynamic acquisition strategy is constructed based on a fair competition lexicon.

[0076] In this embodiment, the fair competition terminology database may include a keyword database related to fair competition review (such as "monopoly agreement", "market dominance", "price discrimination", etc.) and a domain knowledge database, such as a keyword database, a legal basis database, and a case feature database.

[0077] In this embodiment, those skilled in the art can pre-construct fair competition terminology databases for various industries based on keywords related to fair competition review, industry keywords, legal basis terms, case feature terms, etc., and then use these databases to generate corresponding dynamic data collection strategies. After obtaining unified and standardized first data, the dynamic data collection strategies can be used for targeted data collection to obtain second data related to fair competition review for each industry.

[0078] S103: Determine the credibility weight corresponding to the second data, and construct the corresponding fair competition knowledge graph based on the second data.

[0079] In this embodiment, the credibility of each second data point can be scored using dimensions such as the authority of the data source, the freshness of the time, and consistency with other data sources, so as to determine the credibility weight of each second data point. This credibility weight can be used as an attribute label for the second data point, providing a credibility basis for subsequent fair competition review data.

[0080] In this embodiment, data fusion can be performed based on second data with credibility weights to deeply mine the correlation between various data and merge them to construct a corresponding fair competition knowledge graph, providing comprehensive and accurate data basis for subsequent fair competition review.

[0081] In this embodiment, a dynamic adapter library can be used to automatically complete the protocol parsing and format conversion of multi-source heterogeneous data, eliminating the need for manual intervention to adapt to different data sources. This significantly improves data aggregation efficiency, achieves standardized processing of scattered data, and provides unified and standardized basic data for subsequent targeted data collection. A dynamic data collection strategy built based on a fair competition thesaurus enables targeted data capture, accurately filters core information related to fair competition, avoids irrelevant data redundancy, and improves the efficiency, accuracy, and comprehensiveness of data collection. The reliability of the second data can be quantified through credibility weights. A corresponding fair competition knowledge graph can be constructed based on the second data, enabling efficient, comprehensive, and accurate association and fusion of data, providing comprehensive and accurate structured data for subsequent fair competition reviews.

[0082] In one possible implementation, step S101 above, which uses a dynamic adapter library to perform protocol parsing and format conversion on the automatically captured multi-source heterogeneous data to obtain the first data, may include:

[0083] S11: Based on the protocol type of the automatically captured multi-source heterogeneous data, determine the corresponding protocol parsing module in the dynamic adapter library. The dynamic adapter library includes an extensible protocol parsing module.

[0084] S12: Call the protocol parsing module to parse the communication protocol of the multi-source heterogeneous data and obtain the parsed data.

[0085] S13: Based on the format conversion module in the dynamic adapter library, the parsed data is converted to obtain the first data in a unified format.

[0086] In this embodiment, the protocol parsing modules in the dynamic adapter library can correspond one-to-one with the protocol types, and each protocol parsing module can perform communication protocol parsing for the corresponding protocol type.

[0087] In this embodiment, the protocol type may include HTTP, API, FTP, etc. Those skilled in the art can expand the scope of protocol parsing (protocol parsing module type) according to the business scenario of fair competition review, adapt to heterogeneous data of different industries and formats, break the limitation of data source type, and expand the coverage of data collection and processing.

[0088] In this embodiment, the format conversion module can convert various data formats (such as HTML, PDF, JSON, CSV, etc.) into a unified intermediate format, providing unified and standardized basic data for subsequent targeted data collection.

[0089] In this implementation, the dynamic adapter library can include an extensible protocol parsing module, supporting the integration of new protocol parsing modules corresponding to new protocol types. This allows for compatibility with new data sources without reconstructing the overall adaptation architecture, improving the system's adaptability to multi-source data. After data is captured via an interface connection to the data source, the corresponding protocol parsing module in the dynamic adapter library can be called according to the protocol type, eliminating the need for manual intervention to adapt to different data sources and significantly improving data aggregation efficiency. The dynamic adapter library can also include a format conversion module. After parsing the data, the module can output first data in a unified format, eliminating format differences between multi-source data, achieving standardized data integration, avoiding format adaptation costs in subsequent data applications, providing efficient and unified data input for subsequent data processing, and ensuring smooth operation of subsequent processes.

[0090] In one possible implementation, the dynamic acquisition strategy in step S102 above can be constructed in the following way:

[0091] S21: Construct a fair competition terminology database for different industries. The fair competition terminology database includes a keyword database, a legal basis database, and a case feature database. The keyword database includes core keywords and industry keywords, the legal basis database includes legal basis terms, and the case feature database includes case feature terms.

[0092] S22: For each industry, generate a corresponding dynamic data collection strategy based on the industry's fair competition terminology database. The dynamic data collection strategy includes one or more of the following: monitoring data collection rules, price data collection rules, market segmentation data collection rules, and contract text data collection rules.

[0093] In this embodiment, the fair competition terminology database may include, but is not limited to, a keyword database, a legal basis database, and a case feature database. Those skilled in the art can flexibly set it up according to actual needs, and no restrictions are imposed here.

[0094] In this embodiment, the core keywords may include core vocabulary and its extended vocabulary. Those skilled in the art can flexibly set them according to specific industries and actual needs, and no restrictions are imposed here.

[0095] For example, core keywords may include:

[0096] "Monopolistic behavior": ["Monopoly agreement", "horizontal monopoly", "vertical monopoly", "price monopoly"];

[0097] “Market dominance”: [“market dominance”, “abuse of market dominance”, “market share”];

[0098] Unfair competition: ["Price discrimination", "predatory pricing", "tying", "restricted deals"];

[0099] "Concentration of undertakings": ["Concentration of undertakings", "Merger and acquisition control", "Reporting standards"].

[0100] In this embodiment, industry keywords may include industry terms and their extended terms. Those skilled in the art can flexibly set them according to specific industries and actual needs, and no restrictions are imposed here.

[0101] For example, industry keywords can include:

[0102] "Internet": ["Platform exclusivity," "data monopoly," "algorithm collusion"]; "Pharmaceuticals": ["Raw material monopoly," "reverse patent payments"];

[0103] "Finance": ["Interest rate alliance", "market segmentation"].

[0104] In this embodiment, the legal basis term may include legal basis vocabulary and its extended vocabulary. Those skilled in the art can flexibly set it according to specific regulations, and no restrictions are imposed here.

[0105] For example, the terms used to base regulations may include:

[0106] "Anti-Monopoly Law": ["Article 13 Monopoly Agreements", "Article 17 Abuse of Market Dominance"];

[0107] "Anti-Unfair Competition Law": ["Article 6 Commercial Confusion", "Article 12 Unfair Competition on the Internet"].

[0108] In this embodiment, case feature words may include case feature words and their extended words. Those skilled in the art can flexibly set them according to specific cases, and no restrictions are imposed here.

[0109] For example, case feature words may include:

[0110] "Monopoly agreements": ["Price synchronization", "production restrictions", "market segmentation"];

[0111] “Abuse of dominant position”: [“Unfairly high prices”, “refusal to trade”, “discrimination”].

[0112] In this embodiment, the dynamic data collection strategy may include, but is not limited to: monitoring data collection rules, price data collection rules, market segmentation data collection rules, and contract text data collection rules. Those skilled in the art can flexibly set these rules according to actual needs, and no restrictions are imposed here.

[0113] In this implementation, the fair competition terminology can be dynamically updated over time (e.g., adding legal terms, case terms, etc.), and the dynamic data collection strategy is also constantly adjusted accordingly to ensure the timeliness and adaptability of the data collection rules and avoid the omission of effective data due to rule lag.

[0114] In this implementation, a comprehensive thesaurus containing keywords, legal basis, and case features can be constructed by industry, covering core elements of fair competition and industry-specific scenarios. This ensures precise targeting of the data collection scope and avoids cross-industry data confusion. A two-tiered approach of core keywords and industry keywords, combined with legal basis terms and case feature terms, enriches the semantic dimensions of the collected data, ensuring that the collected data conforms to both general fair competition standards and industry-specific competition rules. Corresponding collection strategies are generated for each industry-specific thesaurus, combining rules for monitoring, pricing, market segmentation, and contract texts as needed to achieve differentiated data collection across industries and meet the personalized needs of fair competition analysis in different sectors. Integrating multiple collection rules comprehensively covers key monitoring scenarios for fair competition, accurately capturing core data related to fair competition, reducing irrelevant data redundancy, and improving the efficiency, accuracy, and comprehensiveness of targeted data collection.

[0115] In one possible implementation, determining the credibility weight corresponding to the second data in step S103 above may include:

[0116] S31: Perform structured processing on the structured data in the second data to obtain processed structured data. The structured processing includes one or more of the following: missing value repair, normalization, and deduplication.

[0117] S32: Perform unstructured processing on the unstructured data in the second data to obtain processed unstructured data. The unstructured processing includes one or more of entity error correction and contradiction detection.

[0118] S33: Generate usable data based on the processed structured data and the processed unstructured data.

[0119] S34: Generate the credibility weight for each available data point based on the pre-trained credibility scoring model.

[0120] In this embodiment, the structuring process may include, but is not limited to, missing value repair, normalization, and deduplication. Those skilled in the art can flexibly set these according to actual needs, and no restrictions are imposed here.

[0121] In this embodiment, unstructured processing may include, but is not limited to, entity error correction and contradiction detection. Those skilled in the art can flexibly set these according to actual needs, and no restrictions are imposed here.

[0122] In this implementation, structured data can undergo processes such as missing value repair, normalization, and deduplication to eliminate issues like inconsistent data formats, missing information, and redundant repetition, ensuring data format uniformity and content integrity. For unstructured data, entity error correction and contradiction detection are performed to correct entity errors, eliminate logical inconsistencies, and prevent erroneous or conflicting information from interfering with analysis results, thus improving the usability of unstructured data. The categorized processing mode for structured and unstructured data adapts to the characteristics of different data types, avoiding data corruption or poor results caused by a single processing method, and ensuring that all types of data meet usability standards. The output usable data eliminates noise and defects in the original data, laying a high-quality data foundation for subsequent credibility assessment and knowledge graph construction, enhancing the overall utilization value of the data. The credibility weights of the usable data are automatically generated based on a pre-trained model, objectively quantifying the reliability of the data. These credibility weights can serve as attribute labels for the data, providing a basis for credibility in subsequent fair competition review data.

[0123] In one possible implementation, the credibility scoring model described above can be trained in the following ways:

[0124] S41: Obtain sample data for fair competition review.

[0125] S42: Generate the credibility weight of the sample data based on the data dimensions of the sample data and the corresponding weight ratio of each data dimension. The data dimensions include one or more of the following: data source authority, time freshness, multi-source consistency, content quality, and technical characteristics.

[0126] S43: Generate sample labels for the sample data based on the confidence weight of the sample data.

[0127] S44: Based on the sample data and the sample labels of the sample data, iteratively train the initial classification model to obtain the credibility scoring model.

[0128] In this embodiment, the sample data can be historical data or usable data obtained by processing the above steps S11-S13, S21-S22, S31-S33 (through protocol parsing, formatting, targeted collection, and data structuring / unstructured processing).

[0129] In this embodiment, the classification model can be a random forest model, etc., and no restrictions are imposed here.

[0130] In this embodiment, data dimensions may include, but are not limited to: data source authority, time freshness, multi-source consistency, content quality, and technical characteristics. Those skilled in the art can flexibly set these dimensions according to actual needs, and no restrictions are imposed here.

[0131] In this embodiment, each data dimension can correspond to different weight ratios, evaluation indicators, and calculation methods. For example, Table 1 below shows the weight processing correspondence for each data dimension:

[0132] Table 1

[0133]

[0134] In this implementation, data dimensions can be customized based on the fair competition review scenario, with weight ratios adapted to the scenario requirements. This ensures that the evaluation criteria are highly aligned with business objectives, enhancing the practical reference value of the credibility weights. Data dimensions can cover multiple core dimensions such as data source authority and time freshness. Credibility is calculated by combining the weight ratios of each dimension, avoiding the one-sidedness of single-dimensional evaluation and comprehensively reflecting the reliability of the data. Sample labels are generated based on the credibility weights, providing accurate supervisory signals for model training. This helps the model learn the correlation between data features and credibility, improving scoring accuracy. Iterative training optimizes the initial classification model, continuously correcting prediction biases and gradually adapting the model to the characteristics of fair competition review data. This enhances the model's generalization ability in the target scenario and improves the professionalism of the scoring.

[0135] In one possible implementation, the step S103 above, which involves constructing a corresponding fair competition knowledge graph based on the second data, may include:

[0136] S51: Extract the initial entities corresponding to fair competition from the second data, and use graph embedding technology to map the initial entities to a unified vector space for entity alignment processing to obtain usable entities.

[0137] S52: Determine the entity attributes corresponding to each available entity. Available entities include enterprises, policies, market behaviors, industries, regulations, and impacts.

[0138] S53: Determine the relationships between the available entities and the relationship attributes of each relationship, including competitive relationships, regulatory relationships, compliance relationships, and non-compliance relationships.

[0139] S54: Construct a corresponding fair competition knowledge graph based on each available entity, the entity attributes corresponding to each available entity, the relationships between each available entity, and the relationship attributes of each relationship.

[0140] In this implementation, graph embedding technology (such as TransE) can be used to map entities such as enterprises, policies, and market events to a unified vector space, thereby achieving entity alignment across data sources.

[0141] In this embodiment, the similarity between two entities can be calculated by measuring the Euclidean distance, cosine similarity, etc., thereby eliminating entity duplication and achieving entity alignment.

[0142] In this embodiment, entities may include, but are not limited to: enterprises, policies, market behaviors, industries, regulations, and influences. Those skilled in the art can flexibly set these entities according to actual needs, and no restrictions are imposed here.

[0143] In this embodiment, the relationships may include, but are not limited to: competitive relationships, regulatory relationships, compliance relationships, and violation relationships. Those skilled in the art can flexibly set these relationships according to actual needs, and no restrictions are imposed here.

[0144] In this implementation, the fair competition knowledge graph can link multi-dimensional relationships of "enterprise-behavior-policy-impact".

[0145] For example, Table 2 below shows the alignment criteria between two entities:

[0146] Table 2

[0147]

[0148] In this implementation, graph embedding technology can be used to map initial entities to a unified vector space, achieving precise alignment of entities across data sources, eliminating entity duplication and ambiguity, and ensuring the uniqueness and consistency of entities in the knowledge graph. Entity alignment logic is customized for fair competition scenarios to accurately identify core entities such as enterprises, policies, and regulations, reducing interference from irrelevant entities and improving the quality and usability of core entities in the knowledge graph. Attribute information of all usable entities is comprehensively extracted, while key relationships and attributes such as competition, regulation, and compliance / violation between entities are clarified, constructing a multi-dimensional, three-dimensional knowledge network. It covers the core elements of fair competition, fully presenting the core logic and related paths in the market competition ecosystem, providing comprehensive knowledge coverage for in-depth analysis. Dispersed secondary data is transformed into a structured knowledge graph, intuitively presenting the relationship logic between entities, providing comprehensive and accurate structured data for subsequent fair competition reviews.

[0149] In one possible implementation, after constructing the corresponding fair competition knowledge graph based on the second data in step S103 above, the following may also be included:

[0150] S61: Collect real-time data of dynamic features and determine historical data of dynamic features based on the fair competition knowledge graph.

[0151] S62: A time window mechanism is used to link and integrate real-time data and historical data to obtain integrated data.

[0152] S63: Input the integrated data into the time series model to detect whether dynamic features are abnormal behaviors.

[0153] S64: Determine the data collection weights for dynamic features based on the abnormal behavior detection results of dynamic features.

[0154] S65: Collect multi-source heterogeneous data based on the data collection weights of dynamic features, static features, historical benchmark weights, and industry adjustment factors.

[0155] In this embodiment, dynamic features can be modifiable information in the data, such as stock trading codes, while static features can be information in the data that remains essentially unchanged, such as company names.

[0156] In this embodiment, a sliding time window can be used to link and integrate real-time data and historical data.

[0157] In this embodiment, the time series model can be an LSTM model, etc., and no restrictions are imposed here.

[0158] In this embodiment, when collecting data from multi-source heterogeneous data, the data collection weights can be based on dynamic features, static features, historical benchmarks, and industry adjustment factors. By adjusting these weights, the system can adapt to the needs of the scenario and the characteristics of the industry, balance the requirements of real-time performance and comprehensiveness, and ensure the comprehensiveness and accuracy of the data collection.

[0159] For example, Table 3 below shows the influencing factors and calculation logic for each data collection weight:

[0160] Table 3

[0161]

[0162] Where Wdyn is the data acquisition weight of dynamic features, σ is the activation function, Wd is the learnable dynamic weight matrix, Wsta is the data acquisition weight of static features, Ws is the learnable static weight matrix, ht is the time series feature vector, si is the entity feature vector, [ht‖si] is the vector concatenation operation, bd and bs are bias terms, α is the adjustment coefficient, σhist is the standard deviation of historical data, xt is the time series data at time t, μhist is the mean of historical data, and β is the industry adjustment factor.

[0163] In this implementation, historical data of dynamic features can be retrieved by combining a fair competition knowledge graph with real-time data collection to construct a full-time-series data chain, avoiding the one-sidedness of data from a single time dimension. A time window mechanism is used to integrate time-series data, ensuring the continuity and correlation of data over time, providing complete data context support for abnormal behavior detection, and improving the adaptability of detection scenarios. The integrated data is input into a time-series model to specifically analyze the temporal change patterns of dynamic features, accurately capturing abnormal behaviors that deviate from normal trends, and reducing misjudgments and omissions caused by static analysis. The weights of dynamic feature collection are dynamically adjusted based on the anomaly detection results, prioritizing the collection of abnormal features and reasonably reducing the weight of normal features, avoiding redundant invalid data, and increasing the proportion of high-value data collection. A multi-dimensional collection weight system is constructed by integrating static feature weights, historical benchmark weights, and industry adjustment factors, making the data collection strategy adaptable to scenario requirements and industry characteristics, balancing real-time and comprehensive requirements, reducing the cost of invalid data collection while ensuring that key data is not missed, and improving the overall efficiency of the data processing chain.

[0164] The following describes the data processing method for fair competition review in this application using a specific embodiment.

[0165] In a specific embodiment, the data processing procedure prior to a fair competition review is as follows:

[0166] The first step involves using computing devices to construct fair competition terminology databases for different industries. These databases include a keyword database, a legal basis database, and a case feature database. The keyword database includes core keywords and industry keywords, the legal basis database includes legal basis terms, and the case feature database includes case feature terms. For each industry, a corresponding dynamic data collection strategy is generated based on the industry's fair competition terminology database. This dynamic data collection strategy includes one or more of the following: monitoring data collection rules, price data collection rules, market segmentation data collection rules, and contract text data collection rules.

[0167] The second step involves the computing device connecting to various data sources via interfaces to capture data. Based on the protocol type of the automatically captured multi-source heterogeneous data, the corresponding protocol parsing module in the dynamic adapter library is determined. The protocol parsing module is then called to parse the communication protocol of the multi-source heterogeneous data, resulting in parsed data. Finally, the format conversion module in the dynamic adapter library is used to convert the parsed data to obtain the first data in a unified format.

[0168] The third step involves the computing device performing targeted data acquisition of the first data according to a dynamic acquisition strategy to obtain the second data.

[0169] The fourth step involves the computing device performing structured processing on the structured data in the second data set to obtain processed structured data. The structured processing includes one or more of the following: missing value repair, normalization, and deduplication. The device also performs unstructured processing on the unstructured data in the second data set to obtain processed unstructured data. The unstructured processing includes one or more of the following: entity error correction and contradiction detection. Based on the processed structured data and the processed unstructured data, usable data is generated.

[0170] The fifth step involves the computing device generating a credibility weight for each available data point based on the pre-trained credibility scoring model.

[0171] The sixth step involves the computing device extracting entities corresponding to fair competition from the available data and using graph embedding technology to map the initial entities to a unified vector space for entity alignment to obtain available entities; determining the entity attributes corresponding to each available entity; determining the relationships between each available entity and the relationship attributes of each relationship; and constructing a corresponding fair competition knowledge graph based on each available entity, the entity attributes corresponding to each available entity, the relationships between each available entity, and the relationship attributes of each relationship.

[0172] The seventh step involves the computing device collecting real-time data of dynamic features and determining historical data of dynamic features based on the fair competition knowledge graph; using a time window mechanism to associate and integrate real-time data and historical data to obtain integrated data; and inputting the integrated data into the time series model to detect whether the dynamic features are abnormal behaviors.

[0173] Step 8: The computing device determines the data acquisition weights of the dynamic features based on the abnormal behavior detection results of the dynamic features;

[0174] The ninth step involves the computing device collecting multi-source heterogeneous data based on the data collection weights for dynamic features, static features, historical benchmarks, and industry adjustment factors.

[0175] Below is a simple application example of this data processing procedure:

[0176] (1) Identification of monopoly agreements:

[0177] ① Data collection: We crawled penalty cases from the database of the State Administration for Market Regulation, extracted cooperation agreement clauses from corporate annual reports, and extracted the keyword "price alliance" from news.

[0178] ② Cleaning and alignment: Normalize company names (e.g., merge "XX Group" and "XX Co., Ltd."), and verify time consistency.

[0179] ③ Knowledge graph construction: Link “Company A-Company B-Agreement content-Penalty result” to detect whether there is a horizontal monopoly agreement.

[0180] ④ Output results: Generate a list of risky companies and a relationship graph, and label the data source and credibility score.

[0181] (2) Market dominance assessment:

[0182] ① Data Fusion: Integrate multi-source data such as enterprise revenue (structured tables), industry reports (PDF), and social media sentiment (text).

[0183] ② Time series analysis: Calculate the changes in a company's market share over time to detect whether there is any abuse of dominant position (such as sudden price increases).

[0184] ③ Visualization: Draw a market share trend chart and highlight it in conjunction with relevant clauses of the Anti-Monopoly Law.

[0185] Figure 3 This is a schematic diagram of the structure of a computing device according to an embodiment of this application, as shown below. Figure 3 As shown, the computing device includes: a data acquisition module 31, used to perform protocol parsing and format conversion on automatically captured multi-source heterogeneous data using a dynamic adapter library to obtain first data; and to perform targeted data acquisition on the first data according to a preset dynamic acquisition strategy to obtain second data, wherein the preset dynamic acquisition strategy is constructed based on a fair competition lexicon; and a data fusion module 32, used to determine the credibility weight corresponding to the second data and to construct a corresponding fair competition knowledge graph based on the second data.

[0186] The computing device provided in this application embodiment can execute the technical solution shown in the above method embodiment. Its implementation principle and beneficial effects are similar, and will not be repeated here.

[0187] Figure 4 This is a schematic diagram of the structure of a computing device according to another embodiment of this application, such as... Figure 4 As shown, the computing device includes: a processor 401, and a memory 402 communicatively connected to the processor 401; the memory 402 stores computer execution instructions; the processor 401 executes the computer execution instructions stored in the memory 402 to implement the steps of the data processing method for fair competition review in the above-described method embodiments.

[0188] In the aforementioned computing device, the memory 402 and the processor 401 are electrically connected directly or indirectly to enable data transmission or interaction. For example, these components can be electrically connected to each other via one or more communication buses or signal lines, such as a bus connection. The memory 402 stores computer execution instructions that implement data access control methods, including at least one software functional module that can be stored in the memory 402 in the form of software or firmware. The processor 401 executes various functional applications and data processing by running the software programs and modules stored in the memory 402.

[0189] The memory 402 may be, but is not limited to, Random Access Memory (RAM), Read Only Memory (ROM), Programmable Read-Only Memory (PROM), Erasable Programmable Read-Only Memory (EPROM), Electrically Erasable Programmable Read-Only Memory (EEPROM), etc. The memory 402 stores programs, which are executed by the processor 401 upon receiving execution instructions. Furthermore, the software programs and modules within the memory 402 may also include an operating system, which may include various software components and / or drivers for managing system tasks (e.g., memory management, storage device control, power management, etc.) and can communicate with various hardware or software components to provide an operating environment for other software components.

[0190] Processor 401 can be an integrated circuit chip with signal processing capabilities. The aforementioned processor 401 can be a general-purpose processor, including a Central Processing Unit (CPU), a Network Processor (NP), etc. It can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this application. The general-purpose processor can be a microprocessor or any conventional processor.

[0191] An embodiment of this application also provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, are used to implement the steps of the various method embodiments of this application.

[0192] An embodiment of this application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the various method embodiments of this application.

[0193] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that this application is not limited to the described order of actions, as some steps may be performed in other orders or simultaneously according to this application. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are all optional embodiments, and the actions and modules involved are not necessarily essential to this application.

[0194] It should be further noted that although the steps in the flowchart are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowchart may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the sub-steps or stages of other steps.

[0195] It should be understood that the above-described device embodiments are merely illustrative, and the device of this application can also be implemented in other ways. For example, the division of units / modules in the above embodiments is only a logical functional division, and there may be other division methods in actual implementation. For example, multiple units, modules, or components may be combined, or integrated into another system, or some features may be ignored or not executed.

[0196] Furthermore, unless otherwise specified, the functional units / modules in the various embodiments of this application can be integrated into one unit / module, or each unit / module can exist physically separately, or two or more units / modules can be integrated together. The integrated units / modules described above can be implemented in hardware or as software program modules.

[0197] In the above embodiments, the descriptions of each embodiment have their own emphasis. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments. The technical features of the above embodiments can be combined arbitrarily. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as the combination of these technical features does not contradict each other, it should be considered within the scope of this specification.

[0198] Other embodiments of this application will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of this application that follow the general principles of this application and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this application are indicated by the appended claims.

[0199] It should be understood that this application is not limited to the precise structure described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this application is limited only by the appended claims.

Claims

1. A data processing method for fair competition review, characterized in that, include: A dynamic adapter library is used to perform protocol parsing and format conversion on automatically captured multi-source heterogeneous data to obtain the first data. The first data is collected in a targeted manner according to a preset dynamic collection strategy to obtain the second data. The preset dynamic collection strategy is constructed based on a fair competition lexicon. Determine the credibility weight corresponding to the second data, and construct the corresponding fair competition knowledge graph based on the second data.

2. The data processing method for fair competition review according to claim 1, characterized in that, The first data, obtained by using a dynamic adapter library to perform protocol parsing and format conversion on automatically captured multi-source heterogeneous data, includes: Based on the protocol type of the automatically captured multi-source heterogeneous data, the corresponding protocol parsing module in the dynamic adapter library is determined, and the dynamic adapter library includes an extensible protocol parsing module; The protocol parsing module is invoked to parse the communication protocol of the multi-source heterogeneous data to obtain the parsed data. The parsed data is converted according to the format conversion module in the dynamic adapter library to obtain the first data in a unified format.

3. The data processing method for fair competition review according to claim 1, characterized in that, The dynamic acquisition strategy is constructed in the following way: Construct a fair competition terminology database for different industries. The fair competition terminology database includes a keyword database, a legal basis database, and a case feature database. The keyword database includes core keywords and industry keywords. The legal basis database includes legal basis terms. The case feature database includes case feature terms. For each industry, a corresponding dynamic data collection strategy is generated based on the industry's fair competition terminology database. The dynamic data collection strategy includes one or more of the following: monitoring data collection rules, price data collection rules, market segmentation data collection rules, and contract text data collection rules.

4. The data processing method for fair competition review according to any one of claims 1-3, characterized in that, Determining the credibility weight corresponding to the second data includes: The structured data in the second data is subjected to structured processing to obtain processed structured data. The structured processing includes one or more of missing value repair, normalization, and deduplication. The unstructured data in the second data is subjected to unstructured processing to obtain processed unstructured data. The unstructured processing includes one or more of entity error correction and contradiction detection. Based on the processed structured data and the processed unstructured data, usable data is generated; Based on the pre-trained credibility scoring model, a credibility weight is generated for each piece of available data.

5. The data processing method for fair competition review according to claim 4, characterized in that, The credibility scoring model is trained in the following ways: Obtain sample data for fair competition review; Based on the data dimensions of the sample data and the corresponding weight ratio of each data dimension, a credibility weight of the sample data is generated. The data dimensions include one or more of the following: data source authority, time freshness, multi-source consistency, content quality, and technical characteristics. Based on the credibility weight of the sample data, generate sample labels for the sample data; Based on the sample data and the sample labels of the sample data, the initial classification model is iteratively trained to obtain the credibility scoring model.

6. The data processing method for fair competition review according to any one of claims 1-3, characterized in that, The step of constructing a corresponding fair competition knowledge graph based on the second data includes: Extract the initial entities corresponding to fair competition from the second data, and use graph embedding technology to map the initial entities to a unified vector space to perform entity alignment processing to obtain usable entities; Determine the entity attributes corresponding to each available entity, whereby available entities include enterprises, policies, market behaviors, industries, regulations, and impacts; Determine the relationships between the available entities and the relationship attributes of each relationship, including competitive relationships, regulatory relationships, compliance relationships, and violation relationships; Based on each available entity, the entity attributes corresponding to each available entity, the relationships between each available entity, and the relationship attributes of each relationship, a corresponding knowledge graph of fair competition is constructed.

7. The data processing method for fair competition review according to any one of claims 1-3, characterized in that, After constructing the corresponding fair competition knowledge graph based on the second data, the method further includes: Collect real-time data of dynamic features and determine historical data of the dynamic features based on the fair competition knowledge graph; A time window mechanism is used to associate and integrate the real-time data and the historical data to obtain integrated data; The integrated data is input into a time series model to detect whether the dynamic features constitute abnormal behavior. Based on the abnormal behavior detection results of the dynamic features, the data acquisition weights of the dynamic features are determined; Data collection of multi-source heterogeneous data is carried out based on the data collection weights of dynamic features, static features, historical benchmark weights, and industry adjustment factors.

8. A computing device, characterized in that, include: The data acquisition module is used to perform protocol parsing and format conversion on automatically captured multi-source heterogeneous data using a dynamic adapter library to obtain the first data. The first data is collected in a targeted manner according to a preset dynamic collection strategy to obtain the second data. The preset dynamic collection strategy is constructed based on a fair competition lexicon. The data fusion module is used to determine the credibility weight corresponding to the second data and to construct a corresponding fair competition knowledge graph based on the second data.

9. A computing device, characterized in that, include: A processor, and a memory communicatively connected to the processor; The memory is used to store computer-executed instructions; The processor is configured to execute computer execution instructions stored in the memory, causing the processor to perform the data processing method for fair competition review as described in any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions, which, when executed by a processor, are used to implement the data processing method for fair competition review as described in any one of claims 1-7.

Images