Data processing method, data retrieval method, device, and storage medium

Abstract

Embodiments of the present application provide a data processing method, a data retrieval method, equipment and a storage medium, and relate to the fields of data processing, information retrieval and artificial intelligence. The method comprises: obtaining to-be-processed data, the to-be-processed data comprising business data of at least one business system; performing abnormality detection on the to-be-processed data to determine target data, the target data being business data with quality abnormality; performing weighted calculation processing on the abnormality detection result of the target data in at least one dimension to determine a quality score of the target data; and dynamically adjusting a retrieval weight corresponding to the target data in a retrieval enhancement generation system according to the quality score. The method is used to reduce the interference of abnormal data on retrieval and generation results, and improve the accuracy and reliability of the output of the retrieval enhancement generation system.

Description

Technical Field

[0001] This application relates to the fields of data processing, information retrieval and artificial intelligence, and in particular to a data processing method, a data retrieval method, an apparatus and a storage medium. Background Technology

[0002] In logistics and supply chain scenarios, multi-source data such as waybills, routes, outlets, customer service, and abnormal events are typically integrated into intelligent retrieval and generation systems to support business decisions.

[0003] However, existing solutions often rely on offline cleaning or rule validation for a single data source, which is difficult to adapt to the high-frequency updates of multi-source heterogeneous data. When there are differences in fields, time deviations or semantic inconsistencies for the same business object in different systems, there is often a lack of unified detection capabilities, which leads to a lag in data quality assessment and affects the accuracy and reliability of retrieval and generation results.

[0004] Therefore, how to improve the real-time performance and cross-source consistency of data quality detection in a dynamic environment with multiple data sources, and reduce the impact of low-quality data on the generated search results, has become a technical problem that needs to be solved. Summary of the Invention

[0005] This application provides a data processing method, a data retrieval method, an apparatus, and a storage medium to solve the aforementioned technical problems. This solution addresses business data processing scenarios involving multiple business systems, focusing on the identification of quality anomalies in business data, quality assessment, and coordinated control with the retrieval enhancement generation process. In the context of dynamic updates of multi-source heterogeneous data, it provides targeted processing for data with quality anomalies and adjusts its role in the retrieval enhancement generation system based on data quality, thereby improving the timeliness and effectiveness of data quality detection and downstream application collaboration.

[0006] In a first aspect, embodiments of this application provide a data processing method, including:

[0007] Acquire the data to be processed, which includes business data from at least one business system;

[0008] Perform anomaly detection on the data to be processed to identify the target data, which is the business data with quality anomalies.

[0009] The anomaly detection results of the target data in at least one dimension are weighted and calculated to determine the quality score of the target data.

[0010] Based on the quality score, the retrieval weights corresponding to the target data in the retrieval enhancement generation system are dynamically adjusted.

[0011] In one possible embodiment, the method further includes:

[0012] Determine the metadata tags for the target data. These metadata tags are used to identify the business system and business affiliation information to which the target data belongs.

[0013] Based on metadata tags, the push target is determined from the preset mapping relationship. The push target is the data responsible person associated with the target data.

[0014] Based on the quality score, the retrieval weight of the target data in the retrieval enhancement generation system, and the time-sensitive factor corresponding to the target data, the influencing factor is determined. The influencing factor is used to indicate the potential impact of the target data on the retrieval enhancement generation results.

[0015] Prioritize target data based on impact factors;

[0016] Repair instructions are pushed to the target based on priority.

[0017] In one possible embodiment, after acquiring the data to be processed, the method further includes:

[0018] Identify the field definitions and data structures of data from various business systems within the data to be processed;

[0019] Based on the pre-defined unified semantic model, determine the mapping relationship between data fields of each business system and standard fields;

[0020] Based on the mapping relationship, the fields of data from each business system are aligned to the standard field format under the unified semantic model.

[0021] In one possible embodiment, anomaly detection is performed on the data to be processed to determine the target data, including:

[0022] The description information of the same entity from different business systems in the data to be processed is compared to detect whether there are data conflicts across data sources;

[0023] The semantic information in the data to be processed is parsed to detect whether there are semantic anomalies.

[0024] Perform null checks on required fields in the data to be processed to detect if any fields are missing.

[0025] Perform duplicate identification on the data to be processed to detect whether duplicate records exist;

[0026] Delay determination is performed on the timestamp information in the data to be processed to detect any timeliness anomalies;

[0027] If any of the following conditions exist: data conflict, semantic anomaly, missing field, duplicate record, or timeliness anomaly, the business data will be identified as the target data.

[0028] In one possible embodiment, determining the quality score of the target data includes:

[0029] Determine the first indicator value of the target data in the integrity dimension, which is used to characterize the degree to which fields in the target data are completely filled;

[0030] Determine the second indicator value of the target data in the consistency dimension. The consistency dimension is used to characterize the degree of logical consistency between the target data and data from the same or different sources.

[0031] Determine the third indicator value of the target data in the validity dimension. The validity dimension is used to characterize the degree to which the data values ​​of the target data conform to the preset format specifications, type definitions, or semantic rules.

[0032] Determine the fourth indicator value of the target data in the uniqueness dimension. The uniqueness dimension is used to characterize whether the target data has duplicate records in the business data to which it belongs and the proportion of duplicate records.

[0033] Determine the fifth indicator value of the target data in the timeliness dimension. The timeliness dimension is used to characterize the degree of delay between the update time of the target data and the actual data generation time.

[0034] Determine the weight coefficients for the completeness, consistency, validity, uniqueness, and timeliness dimensions respectively.

[0035] The quality score is obtained by weighting and summing at least one of the first, second, third, fourth, and fifth indicator values ​​and their corresponding weighting coefficients.

[0036] In one possible embodiment, the method further includes:

[0037] Determine the impact threshold corresponding to the impact factor;

[0038] When the impact factor is greater than the impact threshold, a generation control operation is performed on the target data. The generation control operation includes at least one of the following: blocking the target data from participating in the content generation of the retrieval enhancement generation system, and injecting a data confirmation prompt into the information corresponding to the target data in the generation results of the retrieval enhancement generation system.

[0039] In one possible embodiment, repair instructions are pushed to the target based on priority, including:

[0040] Determine the business impact information of the data issues corresponding to the target data. The business impact information includes at least one of the following: impact on transportation timeliness and impact on customer service.

[0041] Determine the difficulty information for repairing the data problem corresponding to the target data. The difficulty information for repair includes at least one of the following: data traceability information and data correctability information.

[0042] Determine the response capability information of the target audience;

[0043] Priorities were adjusted based on information regarding the degree of business impact, the difficulty of remediation, and response capabilities.

[0044] Based on the adjusted priority, send repair instructions to the target.

[0045] Secondly, embodiments of this application provide a data retrieval method, including:

[0046] Retrieve the query statement;

[0047] Based on the retrieval weight, candidate business data corresponding to the query statement is retrieved from the target data source. The retrieval weight is obtained by dynamically adjusting the retrieval weight corresponding to the target data according to the data processing method in the first aspect.

[0048] Based on the candidate business data, determine the target retrieval results corresponding to the query statement.

[0049] Thirdly, embodiments of this application provide a data processing apparatus, including:

[0050] The acquisition module is used to acquire data to be processed, which includes business data from at least one business system.

[0051] The detection module is used to perform anomaly detection on the data to be processed and to identify the target data, which is the business data with quality anomalies.

[0052] The determination module is used to perform weighted calculations on the anomaly detection results of the target data in at least one dimension to determine the quality score of the target data.

[0053] The processing module is used to dynamically adjust the retrieval weights corresponding to the target data in the retrieval enhancement generation system based on the quality score.

[0054] Fourthly, embodiments of this application provide an electronic device, including: a memory and a processor;

[0055] The memory stores instructions that the computer executes;

[0056] The processor executes computer execution instructions stored in memory, causing the processor to perform the data processing method provided above, or causing the processor to perform the data retrieval method provided above.

[0057] Fifthly, embodiments of this application provide a computer-readable storage medium storing computer-executable instructions. When executed by a processor, the computer-executable instructions are used to implement the data processing method provided above, or when executed by a processor, the computer-executable instructions are used to implement the data retrieval method provided above.

[0058] Sixthly, embodiments of this application provide a computer program product, including a computer program that, when executed by a processor, implements the data processing method provided above, or the data retrieval method provided above, when executed by a processor.

[0059] The data processing method, data retrieval method, device, and storage medium provided in this application acquire data to be processed, including business data from at least one business system. They then perform anomaly detection on the data to be processed to identify target data with quality anomalies, further determine the quality score of the target data, and dynamically adjust the retrieval weight corresponding to the target data in the retrieval enhancement generation system based on the quality score. This enables timely identification of low-quality business data and quantification of its quality level in scenarios with dynamically changing multi-source heterogeneous data, thereby reducing the interference of quality anomaly data on retrieval and generation results and improving the accuracy and reliability of the output of the retrieval enhancement generation system. Attached Figure Description

[0060] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0061] Figure 1 This application provides an illustration of an application scenario.

[0062] Figure 2 Flowchart of the data processing method provided in this application Figure 1 ;

[0063] Figure 3 Flowchart of the data processing method provided in this application Figure 2 ;

[0064] Figure 4 A flowchart illustrating the data retrieval method provided in this application;

[0065] Figure 5 A schematic diagram of the data processing apparatus provided in this application;

[0066] Figure 6 A schematic diagram of the structure of the electronic device provided in this application.

[0067] The accompanying drawings illustrate specific embodiments of this application, which will be described in more detail below. These drawings and descriptions are not intended to limit the scope of the concept in any way, but rather to illustrate the concepts of this application to those skilled in the art through reference to particular embodiments. Detailed Implementation

[0068] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.

[0069] Figure 1 This is a schematic diagram illustrating an application scenario provided by this application. The data processing method provided in this application is applied to a scenario in which multiple independent data sources in a logistics transportation scheduling system are accessed in real time through a Retrieval-Augmented Generation (RAG) system. These multiple independent data sources include waybill systems, routing systems, branch system systems, customer service systems, and abnormal event systems, etc., to support the dynamic generation of transportation plans, customer notifications, or abnormal handling suggestions.

[0070] During peak transportation periods, a logistics company's waybill system records frequent updates to order status. However, some branches experience data delays due to system malfunctions. The routing system generates transportation routes based on real-time traffic data, but some routes are incorrectly planned due to data conflicts. These data conflicts include the same order being marked as "in transit" in the waybill system but as "delivered" in the abnormal event system. The customer service system also suffers from missing or semantically contradictory order information due to manual input errors. Semantic contradictions include the "abnormal status" field value exceeding the preset range.

[0071] In such scenarios, multiple business systems continuously generate high-frequency business data. This data needs to serve as the foundation for intelligent retrieval, question and answer generation, scheduling suggestion generation, or customer service response. Therefore, the timeliness and accuracy of data quality assessment are of high importance.

[0072] In existing solutions, a common approach is to first obtain business data from various business systems, and then process the data through offline cleaning, rule validation, or anomaly identification methods tailored to a single system. This includes checking for missing fields, format errors, or local value anomalies, and then providing the processed data for subsequent retrieval or generation processes. This type of solution relies heavily on pre-defined rules and batch processing of historical data. It typically lacks a unified quality assessment mechanism for differences in the state, semantic discrepancies, or inconsistent data timing of the same business object across different systems.

[0073] Because multi-source business data differs in field definitions, update frequencies, and semantic expressions, existing solutions often can only detect anomalies within a single source when faced with real-time changing data, making it difficult to identify quality issues across systems in a timely manner. Even when anomalies are detected, they often remain at the level of data cleaning or alerting, lacking the ability to directly link with the retrieval enhancement generation process. As a result, data with quality anomalies may still participate in retrieval with its original weight.

[0074] In logistics and transportation scheduling, exception handling, and customer notification, the aforementioned deficiencies can further amplify their impact. For example, if the same order is shown as signed for in one system but still as being in transit in another, and this conflict is not promptly identified and reflected in the retrieval stage, the generated results may reference unreliable data. This can lead to biased transportation recommendations, inaccurate customer service responses, or delayed exception handling, ultimately affecting the accuracy and reliability of business decisions.

[0075] In view of this, how to promptly identify target data with quality anomalies after acquiring business data from at least one business system, determine the quality score of the target data based on the anomaly detection results in at least one dimension, and then dynamically adjust the retrieval weight corresponding to the target data in the retrieval enhancement generation system according to the quality score has become an urgent technical problem to be solved.

[0076] To address the aforementioned issues, a data processing method is proposed. This method performs anomaly detection on the data to be processed, filters target data with quality anomalies, and dynamically adjusts the retrieval weight of target data in the retrieval enhancement generation system based on quality scores, thereby controlling the impact of low-quality data. In multi-source data access environments in logistics and supply chains, this method can be deployed in a processing architecture consisting of multiple business systems and a retrieval enhancement generation system, thereby improving the real-time performance of data quality detection and enhancing the accuracy and reliability of retrieval and generation results.

[0077] The technical solution of this application and how the technical solution of this application solves the above-mentioned technical problems are described in detail below with specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments. The embodiments of this application will now be described with reference to the accompanying drawings.

[0078] Figure 2 Flowchart of the data processing method provided in this application Figure 1 ,like Figure 2 As shown, the method includes:

[0079] S101. Obtain the data to be processed, which includes business data from at least one business system.

[0080] In this application, the executing entity can be a data processing server or a data processing node connected to multiple business systems. The business systems, as the source of the data to be processed, are used to continuously output business data, and may specifically include one or more of the following: waybill systems, routing systems, branch system, customer service system, and exception event system. The data to be processed refers to the set of data accessed within the current processing cycle; it can be a real-time data stream generated by the business system, incremental data aggregated according to a time window, or business snapshot data imported in batches.

[0081] In practical implementation, at least one business record output from a business system can be received through conventional data access methods, and the received business data can be sent to subsequent processing as data to be processed. After receiving the business data, the data processing server can maintain the original business attributes of the business data before proceeding to subsequent processing, without changing the basic meaning of the business data at this stage, only performing necessary reception and processing. For batch access data, batches to be processed can be formed according to a preset processing cycle; for real-time data streams, subsequent steps can be initiated after the data arrives. Based on the above analysis, this step, by acquiring business data from at least one business system, forms a set of data to be processed that can be used for subsequent anomaly detection.

[0082] S102. Perform anomaly detection on the data to be processed and determine the target data, which is the business data with quality anomalies.

[0083] In this application, anomaly detection is used to identify business data with quality anomalies from the data to be processed, and to determine this portion of business data as target data. Target data can be business records that exhibit abnormal characteristics in terms of completeness, consistency, validity, uniqueness, and timeliness. Anomaly detection can be performed on data within a single business system or on data related to multiple business systems.

[0084] In practice, the data processing server can perform inspections on the data to be processed based on preset detection rules or conditions to identify any quality anomalies. If anomalies are detected in the business data, the corresponding business data is designated as target data. For ease of subsequent processing, the target data can also be labeled.

[0085] Based on the above analysis, this step can identify business data with quality abnormalities by performing anomaly detection on the data to be processed, and use it as the object of subsequent quality scoring and retrieval weight adjustment.

[0086] S103. Perform weighted calculation on the anomaly detection results of the target data in at least one dimension to determine the quality score of the target data.

[0087] In this application, the quality score is used to quantify the quality of the target data and serves as the basis for subsequent dynamic adjustment of search weights. Dimensions of anomaly detection may include completeness, consistency, validity, uniqueness, and timeliness. Completeness characterizes whether the target data fields are complete; consistency characterizes whether the target data is logically consistent with related data; validity characterizes whether the data values ​​of the target data conform to preset rules; uniqueness characterizes whether there are duplicate records of the target data in its corresponding business data and the proportion of duplicate records; and timeliness characterizes the delay between the update time of the target data and the actual data generation time. The quality score can be determined by any single dimension or by a combination of at least two dimensions.

[0088] In practice, the data processing server can determine at least one anomaly detection result from each dimension for each piece of target data, and then perform a weighted sum based on the determined anomaly detection results and the preset weights corresponding to each dimension to obtain the corresponding quality score. The quality score can be expressed as a normalized score or other comparable score form. The score reflects the quality of the target data; the higher the score, the better the quality of the target data. After determining the quality score, the data processing server can write the quality score into the corresponding quality attribute of the target data for use in the next step.

[0089] Based on the above analysis, this step quantifies the quality of the target data into comparable scores, providing a basis for weight adjustments in the subsequent retrieval enhancement generation system.

[0090] S104. Based on the quality score, dynamically adjust the retrieval weights corresponding to the target data in the retrieval enhancement generation system.

[0091] In this application, the retrieval enhancement generation system is used to receive business data adjusted by quality scoring to participate in the retrieval. Retrieval weight is a parameter that determines the ranking, recall, or reordering of target data during the retrieval phase. A higher weight value indicates a higher priority for the corresponding data in the retrieval results; conversely, a lower weight value indicates a lower priority for the corresponding data in the retrieval results. Dynamic adjustment means that the retrieval weight is not fixed but updated based on the current quality score, causing changes in the participation of target data of different qualities in subsequent retrievals.

[0092] In practice, the data processing server adjusts the retrieval weight of the target data based on its quality score and provides the adjusted retrieval weight to the retrieval enhancement generation system. When the target data has a high quality score, it maintains a high retrieval weight; when the target data has a low quality score, its retrieval weight is reduced to decrease the priority of target data with high quality anomalies in the search results.

[0093] Based on the above analysis, this step applies the quality score to the retrieval weight in the retrieval enhancement generation system, so that the target data participates in subsequent retrieval processing according to its quality, thereby reducing the impact of low-quality data on the retrieval results.

[0094] This application provides a data processing method, comprising: acquiring data to be processed, including business data from at least one business system; performing anomaly detection on the data to be processed to determine target data, wherein the target data is business data with quality anomalies; performing weighted calculation processing on the anomaly detection results of the target data in at least one dimension to determine the quality score of the target data; and dynamically adjusting the retrieval weight corresponding to the target data in the retrieval enhancement generation system based on the quality score. In this application, by acquiring, identifying anomalies, quantifying quality, and adjusting retrieval weights for business data from at least one business system, data with quality anomalies can participate in weight control based on the quality score after entering the retrieval enhancement generation system, thereby reducing the impact of low-quality data on subsequent retrieval results.

[0095] See Figure 3 , Figure 3 Flowchart of the data processing method provided in this application Figure 2 Based on the foregoing embodiments, further, after step S102, the method includes:

[0096] S201. Determine the metadata tag of the target data. The metadata tag is used to identify the business system and business affiliation information to which the target data belongs.

[0097] Metadata tags are data identification information attached to target data. They can characterize which business system the target data comes from, such as the waybill system, routing system, branch system, customer service system, or abnormal event system, as well as the branch, route, customer, or event subject to which the data belongs.

[0098] In practice, after completing the target data filtering, the system reads the metadata fields carried by the target data, or parses the metadata tags from the table name, source interface, branch code, business order number and responsibility domain identifier of the target data, and writes the tags into the tag index table.

[0099] S202. Based on the metadata tags, determine the push target from the preset mapping relationship. The push target is the data responsible person associated with the target data.

[0100] The push target is the data responsible person associated with the target data. This is usually determined by establishing a correspondence between the business system identifier, branch number or attribution code and the specific person in charge, responsible team or work order recipient through a preset mapping relationship.

[0101] In practical implementation, the system queries the data responsible person corresponding to the tag based on the preset mapping relationship. The mapping relationship can be stored in the responsible person configuration library, and the mapping entries can be matched by both business system and attribution information, so as to accurately associate the same type of anomaly with the specific responsible entity.

[0102] S203. Based on the quality score, the retrieval weight of the target data in the retrieval enhancement generation system, and the time-sensitive factor corresponding to the target data, determine the influencing factor. The influencing factor is used to indicate the potential influence of the target data on the retrieval enhancement generation results.

[0103] The impact factor is used to characterize the potential impact of target data on the search enhancement generation results. It is usually formed by combining the quality score, search usage weight and time sensitivity factor. The quality score reflects the degree of data anomaly, the search usage weight reflects the frequency of the data being called or the strength of its contribution in the search enhancement generation system, and the time sensitivity factor reflects the timeliness requirements of the data relative to the current business time.

[0104] For example, the impact factor I can be determined by the following formula:

[0105]

[0106] Where Q is the quality score, and the lower the quality score, the greater the impact factor. W is the weight used by the target data in the retrieval enhancement generation system, which can be dynamically adjusted based on the quality score. T is the time sensitivity factor, which can be preset according to the timeliness requirements of the data. For example, for low-frequency updated data (such as monthly branch inventory reports), T is 0.5, and for high-frequency updated data (such as waybill status updated every second), T is 0.8. W, T, and Q are numbers normalized to the same value range, for example, all within the range [0,1].

[0107] S204. Determine the priority of target data based on impact factors.

[0108] Priority indicates the urgency of addressing a target data issue. Its value can be tiered based on impact factors; the higher the impact factor, the higher the priority. For example, the priority level of target data can be determined using a threshold range for the impact factor.

[0109] S205. Based on priority, push the repair command to the target.

[0110] A repair instruction is a processing notification sent to the push target. The content may include the target data identifier, the anomaly type, the repair requirements, and the processing time limit.

[0111] When target data is determined to be of high priority, the system generates a repair instruction containing an anomaly identifier, the cause of the anomaly, suggested repair content, and a deadline for processing. This instruction is then sent to the target data via the message center, ticket system, email channel, or instant notification channel. If the target data has a lower priority, the repair instruction can be marked as a regular pending task and placed in the subsequent tracking queue. This approach allows for precise association of anomaly information with the responsible party, and tiered outreach based on the impact on the enhanced search results, ensuring that data requiring priority processing enters the repair phase earlier.

[0112] In one possible implementation, a repair instruction is pushed to the target based on priority, including: determining the business impact information of the data problem corresponding to the target data, the business impact information including at least one of the following: transportation timeliness impact information, customer service impact information; determining the repair difficulty information of the data problem corresponding to the target data, the repair difficulty information including at least one of the following: data traceability information, data correctability information; determining the response capability information of the target; adjusting the priority based on the business impact information, repair difficulty information, and response capability information; and sending the repair instruction to the target based on the adjusted priority.

[0113] The information includes: Business Impact Information, which characterizes the intensity of the impact of target data issues on logistics, customer outreach, and customer service handling; Transportation Timeliness Impact Information, which can be determined based on arrival time deviations, route recalculations, or conflicts between pickup and receipt statuses caused by abnormal data; and Customer Service Impact Information, which can be determined based on the number of complaint tickets, the number of failed notifications, or deviations in customer service responses. Repair Difficulty Information, which characterizes the processing complexity required to recover from a target data issue; Data Traceability Information, which can be determined by checking whether the data has a source system identifier, change path, and responsible person association records; and Data Correctability Information, which can be determined by judging whether fields allow write-back, whether association verification is complete, and whether repair requires cross-system synchronization. Target Response Capability Information, which characterizes the processing capability of the data responsible person or handling team to receive and complete repair instructions, can be determined based on historical average response time, ticket closure rate, and the current amount of pending tasks.

[0114] In practical implementation, after identifying target data, the system extracts the associated anomaly type, business scenario, and responsible entity from the metadata and maps them to corresponding business impact assessment units, repair difficulty assessment units, and response capability assessment units. Business impact information and repair difficulty information can be quantified using preset scoring models. These models can be segmented scoring based on rule thresholds or continuous scoring based on weighted functions. Response capability information is generated based on the average confirmation time and processing completion rate of the target within a preset time window. The system then inputs these scores into a priority adjustment model to adjust the original priority upwards or downwards, outputting an adjusted priority queue. Finally, it sends repair instructions containing anomaly descriptions, repair suggestions, and confirmation requirements to the corresponding target according to the queue order.

[0115] This approach reorders the push of repair tasks by simultaneously incorporating three types of information: the degree of business impact, the difficulty of repair, and the responsiveness. This prioritizes high-impact issues that can be quickly resolved, ensuring that the responsible parties are reached first. Because the order in which repair instructions are sent aligns with the actual impact of the anomaly on the business, and tasks can be allocated based on handling capabilities, the efficiency of repair resource utilization is improved, and the timeliness of handling critical data anomalies is enhanced.

[0116] Using the above methods, the business ownership of the target data can be accurately identified, the responsible parties can be targeted, and the order of repair triggering is consistent with the degree of impact of the data on the search enhancement results. Therefore, high-impact anomalies can be processed first, and the data status used in subsequent search and generation processes can be more consistent.

[0117] Based on the aforementioned embodiments, after acquiring the data to be processed, the method further includes: identifying the field definitions and data structures of the data from each business system in the data to be processed; determining the mapping relationship between the data fields of each business system and standard fields according to a preset unified semantic model; and aligning the fields of the data from each business system to the standard field format under the unified semantic model according to the mapping relationship.

[0118] Among them, field definition is used to describe the meaning, value boundary and usage rules of fields in each business system, data structure is used to describe the organization form and field combination method of data in each business system, unified semantic model is used to uniformly express and align data fields of different business systems, standard field is used to carry the unified field form of data of each business system under the unified semantic model, and mapping relationship is used to establish a one-to-one or many-to-one correspondence between fields of each business system and standard field.

[0119] In actual processing, the data to be processed may include structured records from different business systems, such as waybills, routes, branch locations, customer service records, and abnormal events. After receiving the data, the system first parses the field names, field descriptions, field types, hierarchical relationships, and inter-record relationships of each record to identify field definitions and data structures. When the data to be processed exists in the form of table structures, JSON (JavaScript Object Notation) structures, or message bodies, the semantics of the fields can be extracted through field metadata, interface documentation, or data dictionaries, and the field descriptions of the same business object in different systems can be merged.

[0120] After completing field identification, the system establishes a standard field set based on a preset unified semantic model. This standard field set corresponds to unified expressions such as order identifier, shipping status, timestamp, site code, and exception type. Mapping rules from field to standard field are generated based on field semantics, value constraints, and business affiliation. Fields with the same semantics but different names can be mapped to the same standard field; for a business field carrying multiple semantics, it can be split and mapped to multiple standard fields; and for multiple business fields representing the same semantics, they can be merged and mapped to a single standard field.

[0121] After obtaining the mapping relationship, the system renames, converts, normalizes, expands, or merges the original fields according to the mapping rules. This aligns the data from various business systems to the standard field format under a unified semantic model and outputs the standardized data to be processed. After this processing, similar business data from different business systems can be compared, retrieved, and judged for quality within the same semantic space, thus providing consistent data input for subsequent anomaly detection, quality scoring, and retrieval weight adjustment.

[0122] In one possible implementation, anomaly detection is performed on the data to be processed to determine the target data. This includes: comparing descriptive information of the same entity from different business systems in the data to be processed to detect whether there are cross-data source conflicts; parsing the semantic information in the data to be processed to detect whether there are semantic anomalies; performing null value validation on the required fields in the data to be processed to detect whether there are missing fields; identifying duplicate records in the data to be processed to detect whether there are duplicate records; and determining the delay of timestamp information in the data to be processed to detect whether there are timeliness anomalies. If any one of the following exists—data conflict, semantic anomaly, missing fields, duplicate records, or timeliness anomalies—the business data is determined as the target data.

[0123] In actual processing, after receiving pending data from waybills, routing, branch offices, customer service, and abnormal event business systems, the system first performs association matching on the identification information corresponding to the same business object. Then, it compares the descriptions of the same order, branch office, or abnormal event in each system item by item. If inconsistencies are found in the receipt status, transportation status, abnormal conclusion, or attribution information, it is recorded as a cross-data source data conflict. At the same time, the system performs semantic parsing on status codes, event descriptions, text tags, or coded fields, and determines whether they exceed the allowed value range or conflict with the meaning of other fields based on a preset semantic rule base, thereby identifying semantics. An anomaly is identified; next, null checks are performed on required fields. If empty strings, empty objects, or fields not submitted exist, the field is considered missing. Then, duplicate records of the same business object within a single data source or across data sources are identified. Based on preset combinations of primary key fields or key business fields, it is checked whether two or more identical records exist; if so, they are considered duplicate records. Finally, the difference between the timestamp field in the data record and the occurrence time of the corresponding actual business event is calculated to check whether the data update time lags behind the actual occurrence time of the business event. If the time difference exceeds a preset time tolerance threshold, it is considered insufficient timeliness. Once any anomaly is identified, the corresponding business data is marked as target data and output to the subsequent quality scoring and retrieval weight adjustment modules.

[0124] This anomaly detection mechanism incorporates cross-system consistency, semantic validity, field completeness, data uniqueness, and data timeliness into its judgment scope, enabling target data to be identified in a timely manner before entering the retrieval enhancement generation system. For data with conflicting descriptions of the same business object across different business systems, its quality anomalies can be directly intercepted; data with semantic expressions deviating from preset rules can be identified at the text or encoding level; data with missing required fields can be marked at the completeness level; data with duplicate records of the same business object within or across systems can be accurately detected at the record level, preventing redundant information from interfering with subsequent generation; and data whose update time lags behind the actual business event time can be promptly detected at the time dimension, ensuring that expired or outdated data does not participate in subsequent generation. Therefore, business data with potential quality risks can be uniformly identified as target data, providing accurate input for subsequent quality scoring and retrieval control.

[0125] In one possible implementation, determining the quality score of the target data includes: determining a first indicator value for the target data in the integrity dimension, which characterizes the degree to which fields in the target data are fully filled; determining a second indicator value for the target data in the consistency dimension, which characterizes the degree of logical consistency between the target data and data from the same or different sources; determining a third indicator value for the target data in the validity dimension, which characterizes the degree to which the data values ​​of the target data conform to preset format specifications, type definitions, or semantic rules; determining a fourth indicator value for the target data in the uniqueness dimension, which characterizes whether the target data contains duplicate records in its associated business data and the proportion of duplicate records; determining a fifth indicator value for the target data in the timeliness dimension, which characterizes the degree of delay between the update time of the target data and the actual data generation time; determining the weight coefficients corresponding to the integrity, consistency, validity, uniqueness, and timeliness dimensions respectively; and performing a weighted sum based on at least one of the first, second, third, fourth, and fifth indicator values ​​and their corresponding weight coefficients to obtain the quality score.

[0126] In this application, the integrity dimension is used to quantify the filling status of key fields in the target data. The first indicator value can be calculated based on the number of empty values, the number of valid filled values, and the total number of fields in the required fields. The consistency dimension is used to compare the status, numerical, and semantic correspondence between the target data and data from the same source or across sources. The second indicator value can be determined based on the matching ratio or conflict degree of consistent fields. The validity dimension is used to validate the format, type, and semantic rules of the field values ​​in the target data. The third indicator value can be calculated based on the percentage of fields that pass the validation or the percentage of non-compliant fields. The uniqueness dimension is used to quantify the degree of duplication of records of the same business object in the target data. The fourth indicator value can be determined based on the proportion of duplicate records to the total number of records or the unique matching ratio based on the primary key field. The timeliness dimension is used to quantify the degree of delay between the timestamp information in the target data and the corresponding actual business event occurrence time. The fifth indicator value can be calculated based on the proportion of records whose time difference is within the preset tolerance threshold or the proportion of records that have timed out. Weighting coefficients reflect the relative importance of each dimension in the current business scenario. They can be preset or configured based on the business attributes of order, route, branch, customer service, or abnormal event data. The sum of the weights for completeness, consistency, validity, uniqueness, and timeliness is one. After uniform normalization, each indicator value is weighted and summed with its corresponding weighting coefficient to obtain a quality score. The quality score is used to characterize the overall quality level of the target data.

[0127] In practical applications, the first indicator value can be calculated using a field fill rate model, which is obtained by dividing the number of filled key fields in the target data by the total number of key fields. The second indicator value can be calculated using a consistency comparison model, which involves aligning and comparing the status, time, or identifier fields of the same business object across different systems. The third indicator value can be calculated using a rule validation model, which involves matching field values ​​against preset format specifications, type definitions, or semantic rules item by item. The fourth indicator value can be calculated using a duplicate detection model, which involves grouping and statistically analyzing records of the same business object based on preset primary key field combinations or key business field combinations, and determining the indicator value by the proportion of duplicate records to the total number of records. The fifth indicator value can be calculated using a timeliness judgment model, which involves calculating the difference between the timestamp field in the target data and the corresponding actual business event occurrence time, and using the proportion of records whose difference is within a preset tolerance threshold to the total number of records as the indicator value. Weighting coefficients can be pre-issued by the business configuration center or adjusted based on business type, data source reliability, and retrieval frequency. This application does not limit the specific values ​​of the weighting coefficients.

[0128] For example, the quality score Q can be determined by the following formula:

[0129]

[0130] Wherein, P is the first indicator value of the target data in the completeness dimension; C is the second indicator value of the target data in the consistency dimension; V is the third indicator value of the target data in the validity dimension; U is the fourth indicator value of the target data in the uniqueness dimension; and L is the fifth indicator value of the target data in the validity dimension. These are the weighting coefficients corresponding to each indicator value, and their sum is 1.

[0131] This quality scoring system uniformly measures and weights completeness, consistency, validity, uniqueness, and timeliness, mapping missing, conflicting, rule-violating, duplicated, and information-lagging issues in multi-source business data to a single score result. This allows the quality status of the target data to be directly used for subsequent retrieval weight adjustment. Because the scoring process simultaneously covers five dimensions—field completion, cross-source logical consistency, format and semantic compliance, record uniqueness, and data freshness—it provides a more comprehensive characterization of multi-source data quality in logistics and supply chain scenarios. In particular, the introduction of the timeliness dimension effectively quantifies real-time data issues unique to logistics, such as delayed route updates and delayed delivery status reporting, better aligning with the stringent real-time requirements of express delivery logistics. Using this implementation method, the quality level of the target data can be quantified into a unified score, providing a direct basis for weight adjustment in the retrieval enhancement generation system. This controls the impact of quality anomalies on retrieval results and improves the accuracy and reliability of subsequent generated results.

[0132] Based on the aforementioned embodiments, the method further includes: determining the impact threshold corresponding to the impact factor; when the impact factor is greater than the impact threshold, performing a generation control operation on the target data, the generation control operation including at least one of the following: blocking the target data from participating in the content generation of the retrieval enhancement generation system, and injecting a data confirmation prompt into the generation results of the retrieval enhancement generation system for the information corresponding to the target data.

[0133] In this implementation, the impact threshold is used as a boundary value to characterize whether the degree of impact needs to be controlled. The impact threshold can be preset based on the risk of misuse of target data in historical generation tasks, business timeliness requirements, and manual verification costs, and can be stored in the parameter configuration module. It is loaded into the processing unit by the management terminal during system deployment. The comparison between the impact factor and the impact threshold can be completed by the rule engine or the threshold judgment module. When the value corresponding to the impact factor is higher than the impact threshold, the generation control module is triggered to implement restrictive processing on the target data.

[0134] The generation control operation can be executed collaboratively by the retrieval control submodule and the prompt injection submodule. The retrieval control submodule writes the target data's identification information into a blacklist or low-confidence table, preventing the target data from being used as a basis for answer concatenation in subsequent vector retrieval, keyword recall, or re-ranking processes. The prompt injection submodule inserts a "Data Pending Confirmation" prompt corresponding to the target data into the generated results. This prompt can be directly associated with the target data's business number, field name, or source system identifier, so that the output results simultaneously indicate the need for manual verification when presenting relevant information. If the system is configured in prompt-only mode, the target data can still remain in the search candidates, but a confidence level is added during the generation stage, and a prompt statement is embedded in the final output text.

[0135] In practical applications, the impact threshold can be configured according to different business scenarios. For example, different thresholds can be set for data such as transportation status, receipt status, or abnormal events to adapt to their sensitivity to the generated results. The threshold configuration method and the implementation form of the control module in this application are not limited to a specific hardware structure. In practical applications, other models of this component can also be selected, and this application does not limit this.

[0136] By initiating generation control operations when the impact factor exceeds the impact threshold, the system can limit the impact of quality abnormal data on the generated search enhancement results to a controllable range, and provide confirmation prompts for relevant outputs when necessary, thereby ensuring that the generated results are consistent with the data reliability status and reducing the propagation of errors caused by low-quality data directly participating in content generation.

[0137] This application also provides a data retrieval method applied to a retrieval enhancement generation system in a logistics business scenario. The retrieval weights corresponding to target data in the retrieval enhancement generation system are dynamically adjusted according to the aforementioned data processing method. The data retrieval method is described below with reference to the accompanying drawings.

[0138] Figure 4 A flowchart illustrating the data retrieval method provided in this application. (For example...) Figure 4 As shown, the method includes:

[0139] S301, Obtain the query statement.

[0140] A query statement refers to text information entered by a user in natural language or structured query format to express their information retrieval needs. For example, the system can receive user-inputted text or query statements through an interactive interface provided by the retrieval enhancement generation system, or it can receive query requests from upstream business systems via an Application Programming Interface (API) to determine the query statement. After obtaining the query statement, the system can also preprocess it, such as performing word segmentation, stop word removal, and entity recognition, to extract key information from the query statement, facilitating subsequent retrieval processes.

[0141] S302. Based on the retrieval weight, retrieve the candidate business data corresponding to the query statement from the target data source.

[0142] The retrieval weights are obtained by dynamically adjusting the retrieval weights corresponding to the target data according to the data processing methods described above. Specifically, in the data preparation stage, the system acquires business data to be processed, including data from at least one business system. Anomaly detection is performed on the data to be processed to identify target data with quality anomalies. The anomaly detection results for the target data in at least one dimension are weighted to determine a quality score for the target data. Based on the quality score, the retrieval weights corresponding to the target data in the retrieval enhancement generation system are dynamically adjusted. After this dynamic adjustment, each data item in the retrieval enhancement generation system has its own retrieval weight; data items with higher quality scores correspond to higher retrieval weights, and data items with lower quality scores correspond to lower retrieval weights. Therefore, when performing a retrieval operation, the retrieval enhancement generation system can retrieve data based on the pre-adjusted retrieval weights.

[0143] The target data source refers to the database used to store business data in the retrieval enhancement generation system. In a logistics business scenario, the target data source may include one or more of the following: waybill system database, routing system database, branch system database, customer service system database, and exception event system database.

[0144] Recall refers to the process of retrieving and obtaining data items associated with a query statement from a target data source. Specifically, the system first vectorizes the query statement, converting it into a query vector; simultaneously, each business data in the target data source is pre-vectorized and stored as a vector index. The system calculates the similarity (e.g., cosine similarity or Euclidean distance) between the query vector and each business data vector, and combines this with dynamically adjusted retrieval weights corresponding to each business data to weight the similarity scores, obtaining a comprehensive score for each business data; then, a preset number of business data are selected as candidate business data in descending order of comprehensive scores. The introduction of retrieval weights ensures that higher-quality data items have a higher recall priority during the recall process, while data items with abnormal quality have a lower recall priority, thus guaranteeing data quality. In other implementations, recall can also employ keyword matching, hybrid retrieval methods, etc., and this application does not impose specific limitations on these. In hybrid retrieval methods, dynamically adjusted retrieval weights can be used to adjust the fusion ratio of different retrieval methods in the final comprehensive score.

[0145] S303. Based on the candidate business data, determine the target retrieval result corresponding to the query statement.

[0146] Retrieval results refer to the output content containing the information required by the query, generated in response to the query statement. In the retrieval enhancement generation system, determining the target retrieval results specifically includes: using the recalled candidate business data as contextual information, and inputting it together with the query statement into the Large Language Model (LLM). The Large Language Model then generates an answer or report in natural language form based on the factual information provided by the candidate business data.

[0147] In one possible implementation, the recalled candidate business data can be reordered before generating the target retrieval results. Specifically, the candidate business data are sorted from high to low according to their comprehensive score (which incorporates dynamically adjusted retrieval weights), and a preset number of the top-ranked candidate business data are selected for generating the target retrieval results, thereby controlling the context length input to the large language model while ensuring data quality.

[0148] This embodiment obtains the query statement, recalls candidate business data from the target data source according to the retrieval weight, and generates target retrieval results based on the recalled candidate business data. This allows the data quality assessment results to directly affect the online retrieval process of the retrieval enhancement generation system, realizing a closed-loop linkage between data quality governance and the retrieval enhancement generation system. This effectively improves the retrieval accuracy and reliability of the generated content in logistics business scenarios.

[0149] Figure 5A schematic diagram of the data processing apparatus provided in this application is shown below. Figure 5 As shown, the data processing device 20 provided in this embodiment includes:

[0150] Acquisition module 21 is used to acquire data to be processed, which includes business data from at least one business system;

[0151] The detection module 22 is used to perform anomaly detection on the data to be processed and to determine the target data, which is the business data with quality anomalies.

[0152] Module 23 is used to perform weighted calculation on the anomaly detection results of the target data in at least one dimension to determine the quality score of the target data.

[0153] The processing module 24 is used to dynamically adjust the retrieval weights corresponding to the target data in the retrieval enhancement generation system based on the quality score.

[0154] In one possible implementation, the data processing device 20 is further used for:

[0155] Determine the metadata tags for the target data. These metadata tags are used to identify the business system and business affiliation information to which the target data belongs.

[0156] Based on metadata tags, the push target is determined from the preset mapping relationship. The push target is the data responsible person associated with the target data.

[0157] Based on the quality score, the retrieval weight of the target data in the retrieval enhancement generation system, and the time-sensitive factor corresponding to the target data, the influencing factor is determined. The influencing factor is used to indicate the potential impact of the target data on the retrieval enhancement generation results.

[0158] Prioritize target data based on impact factors;

[0159] Repair instructions are pushed to the target based on priority.

[0160] In one possible implementation, the data processing device 20 is further used for:

[0161] Identify the field definitions and data structures of data from various business systems within the data to be processed;

[0162] Based on the pre-defined unified semantic model, determine the mapping relationship between data fields of each business system and standard fields;

[0163] Based on the mapping relationship, the fields of data from each business system are aligned to the standard field format under the unified semantic model.

[0164] In one possible implementation, the detection module 22 is specifically used for:

[0165] The description information of the same entity from different business systems in the data to be processed is compared to detect whether there are data conflicts across data sources;

[0166] The semantic information in the data to be processed is parsed to detect whether there are semantic anomalies.

[0167] Perform null checks on required fields in the data to be processed to detect if any fields are missing.

[0168] Perform duplicate identification on the data to be processed to detect whether duplicate records exist;

[0169] Delay determination is performed on the timestamp information in the data to be processed to detect any timeliness anomalies;

[0170] If any of the following conditions exist: data conflict, semantic anomaly, missing field, duplicate record, or timeliness anomaly, the business data will be identified as the target data.

[0171] In one possible implementation, module 23 is specifically used for:

[0172] Determine the first indicator value of the target data in the integrity dimension, which is used to characterize the degree to which fields in the target data are completely filled;

[0173] Determine the second indicator value of the target data in the consistency dimension. The consistency dimension is used to characterize the degree of logical consistency between the target data and data from the same or different sources.

[0174] Determine the third indicator value of the target data in the validity dimension. The validity dimension is used to characterize the degree to which the data values ​​of the target data conform to the preset format specifications, type definitions, or semantic rules.

[0175] Determine the fourth indicator value of the target data in the uniqueness dimension. The uniqueness dimension is used to characterize whether the target data has duplicate records in the business data to which it belongs and the proportion of duplicate records.

[0176] Determine the fifth indicator value of the target data in the timeliness dimension. The timeliness dimension is used to characterize the degree of delay between the update time of the target data and the actual data generation time.

[0177] Determine the weight coefficients for the completeness, consistency, validity, uniqueness, and timeliness dimensions respectively.

[0178] The quality score is obtained by weighting and summing at least one of the first, second, third, fourth, and fifth indicator values ​​and their corresponding weighting coefficients.

[0179] In one possible implementation, the data processing device 20 is further used for:

[0180] Determine the impact threshold corresponding to the impact factor;

[0181] When the impact factor is greater than the impact threshold, a generation control operation is performed on the target data. The generation control operation includes at least one of the following: blocking the target data from participating in the content generation of the retrieval enhancement generation system, and injecting a data confirmation prompt into the information corresponding to the target data in the generation results of the retrieval enhancement generation system.

[0182] In one possible implementation, the data processing device 20 is further used for:

[0183] Determine the business impact information of the data issues corresponding to the target data. The business impact information includes at least one of the following: impact on transportation timeliness and impact on customer service.

[0184] Determine the difficulty information for repairing the data problem corresponding to the target data. The difficulty information for repair includes at least one of the following: data traceability information and data correctability information.

[0185] Determine the response capability information of the target audience;

[0186] Priorities were adjusted based on information regarding the degree of business impact, the difficulty of remediation, and response capabilities.

[0187] Based on the adjusted priority, send repair instructions to the target.

[0188] The data processing device provided in this embodiment can execute the data processing method provided in the above method embodiment. Its implementation principle and technical effect are similar, and will not be described in detail here.

[0189] Figure 6 A schematic diagram of the structure of the electronic device provided in this application. Figure 6 As shown, the electronic device 30 provided in this embodiment includes at least one processor 31 and a memory 32. Optionally, the device 30 also includes a communication component 33. The processor 31, memory 32, and communication component 33 are connected via a bus.

[0190] In the specific implementation process, at least one processor 31 executes computer execution instructions stored in memory 32, causing at least one processor 31 to execute the above-mentioned data processing method or data retrieval method.

[0191] The specific implementation process of processor 31 can be found in the above method embodiments, and its implementation principle and technical effect are similar. It will not be repeated here.

[0192] In the above embodiments, it should be understood that the processor can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), etc. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in this invention can be directly implemented by a hardware processor, or implemented by a combination of hardware and software modules within the processor.

[0193] The memory may include random access memory (RAM) and may also include non-volatile memory (NVM), such as at least one disk storage device.

[0194] The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of illustration, the buses shown in the accompanying drawings are not limited to a single bus or a single type of bus.

[0195] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the above-described data processing method or data retrieval method.

[0196] This application also provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, implement the aforementioned data processing method or data retrieval method.

[0197] The aforementioned readable storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk. The readable storage medium can be any available medium accessible to a general-purpose or special-purpose computer.

[0198] An exemplary readable storage medium is coupled to a processor, enabling the processor to read information from and write information to the readable storage medium. Of course, the readable storage medium can also be a component of the processor. The processor and the readable storage medium can reside in an application-specific integrated circuit (ASIC). Alternatively, the processor and the readable storage medium can exist as discrete components in the device.

[0199] The division of units is merely a logical functional division; in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be indirect coupling or communication connection through some interfaces, devices, or units, and may be electrical, mechanical, or other forms.

[0200] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0201] In addition, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0202] If a function is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory, random access memory, magnetic disks, or optical disks.

[0203] Those skilled in the art will understand that all or part of the steps of the above-described method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When executed, the program performs the steps of the above-described method embodiments; and the aforementioned storage medium includes various media capable of storing program code, such as ROM, RAM, magnetic disks, or optical disks.

[0204] Finally, it should be noted that other embodiments of the invention will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This invention is intended to cover any variations, uses, or adaptations of the invention that follow the general principles of the invention and include common knowledge or customary techniques in the art not disclosed herein, and is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of the invention is limited only by the appended claims.

Claims

1. A data processing method, characterized in that, include: Acquire data to be processed, wherein the data to be processed includes business data from at least one business system; Anomaly detection is performed on the data to be processed to determine the target data, which is business data with quality anomalies; The anomaly detection results of the target data in at least one dimension are weighted and calculated to determine the quality score of the target data. Based on the quality score, the retrieval weights corresponding to the target data in the retrieval enhancement generation system are dynamically adjusted.

2. The method according to claim 1, characterized in that, The method further includes: Determine the metadata tag of the target data, the metadata tag being used to identify the business system and business affiliation information to which the target data belongs; Based on the metadata tags, the push target is determined from the preset mapping relationship, and the push target is the data responsible person associated with the target data; Based on the quality score, the retrieval weight of the target data in the retrieval enhancement generation system, and the time-sensitive factor corresponding to the target data, an impact factor is determined. The impact factor is used to indicate the potential impact of the target data on the retrieval enhancement generation results. The priority of the target data is determined based on the impact factors; Repair instructions are pushed to the target device according to the priority.

3. The method according to claim 1, characterized in that, After acquiring the data to be processed, the method further includes: Identify the field definitions and data structures of each business system's data in the data to be processed; Based on the pre-defined unified semantic model, determine the mapping relationship between data fields of each business system and standard fields; Based on the mapping relationship, the fields of data from each business system are aligned to the standard field format under the unified semantic model.

4. The method according to claim 1, characterized in that, The step of performing anomaly detection on the data to be processed and determining the target data includes: The description information of the same entity from different business systems in the data to be processed is compared to detect whether there are data conflicts across data sources; The semantic information in the data to be processed is parsed to detect whether there are any semantic anomalies; Null value validation is performed on the required fields in the data to be processed to detect whether any fields are missing; The data to be processed is subjected to duplicate identification to detect whether duplicate records exist; The timestamp information in the data to be processed is delayed to detect whether there is any timeliness abnormality; If any of the following conditions exist: data conflict, semantic anomaly, missing field, duplicate record, or timeliness anomaly, the business data will be identified as the target data.

5. The method according to claim 1, characterized in that, Determining the quality score of the target data includes: Determine the first indicator value of the target data in the integrity dimension, whereby the integrity dimension is used to characterize the degree to which fields in the target data are fully filled; Determine a second indicator value for the target data in the consistency dimension, wherein the consistency dimension is used to characterize the degree of logical consistency between the target data and data from the same or different sources; Determine the third indicator value of the target data in the validity dimension. The validity dimension is used to characterize the degree to which the data value of the target data conforms to the preset format specifications, type definitions, or semantic rules. Determine the fourth indicator value of the target data in the uniqueness dimension. The uniqueness dimension is used to characterize whether the target data has duplicate records in the business data to which it belongs and the proportion of duplicate records. The fifth indicator value of the target data in the timeliness dimension is determined, and the timeliness dimension is used to characterize the degree of delay between the update time of the target data and the actual data generation time; Determine the weight coefficients for the completeness dimension, the consistency dimension, the validity dimension, the uniqueness dimension, and the timeliness dimension respectively; The quality score is obtained by weighted summation based on at least one of the first, second, third, fourth, and fifth indicator values ​​and their corresponding weighting coefficients.

6. The method according to claim 2, characterized in that, The method further includes: Determine the impact threshold corresponding to the impact factor; When the influence factor is greater than the influence threshold, a generation control operation is performed on the target data. The generation control operation includes at least one of the following: blocking the target data from participating in the content generation of the retrieval enhancement generation system, and injecting a data confirmation prompt into the generation results of the retrieval enhancement generation system for the information corresponding to the target data.

7. The method according to claim 2, characterized in that, The step of pushing repair instructions to the target according to the priority includes: Determine the business impact information of the data problem corresponding to the target data, wherein the business impact information includes at least one of the following: transportation timeliness impact information and customer service impact information; Determine the repair difficulty information of the data problem corresponding to the target data, wherein the repair difficulty information includes at least one of the following: data traceability information and data correctability information; Determine the response capability information of the target device; The priority is adjusted based on the information regarding the degree of business impact, the information regarding the difficulty of repair, and the information regarding response capability; The repair instruction is sent to the target device according to the adjusted priority.

8. A data retrieval method, characterized in that, include: Retrieve the query statement; Candidate business data corresponding to the query statement are retrieved from the target data source according to the retrieval weight, wherein the retrieval weight is obtained by dynamically adjusting the retrieval weight corresponding to the target data according to any one of the data processing methods according to claims 1-7; Based on the candidate business data, determine the target retrieval result corresponding to the query statement.

9. A data processing apparatus, characterized in that, include: The acquisition module is used to acquire data to be processed, which includes business data from at least one business system. The detection module is used to perform anomaly detection on the data to be processed and determine the target data, which is business data with quality anomalies; The determination module is used to perform weighted calculation processing on the anomaly detection results of the target data in at least one dimension to determine the quality score of the target data; The processing module is used to dynamically adjust the retrieval weights corresponding to the target data in the retrieval enhancement generation system based on the quality score.

10. An electronic device, characterized in that, include: Memory, processor; The memory stores computer-executed instructions; The processor executes computer execution instructions stored in the memory, causing the processor to perform the data processing method as described in any one of claims 1-7, or causing the processor to perform the data retrieval method as described in claim 8.

11. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions, which, when executed by a processor, are used to implement the data processing method as described in any one of claims 1-7, or, when executed by a processor, are used to implement the data retrieval method as described in claim 8.

12. A computer program product, characterized in that, The computer program, when executed by a processor, implements the data processing method according to any one of claims 1-7, or when executed by a processor, implements the data retrieval method according to claim 8.

Images