A data preprocessing and loading control system for a commercial AI model

Abstract

The application discloses a kind of data preprocessing and loading control system of commercial AI model, including following steps: acquisition module is used to obtain current commercial data batch and corresponding data;State extraction module is used to extract feature and construct current batch query state;Memory construction module is used to form multiple historical memory units and construct historical data processing memory field;Association retrieval module is used to obtain candidate memory unit set based on improved Hopfield network;Trajectory reconstruction module is used to generate target preprocessing action chain and target loading action chain;Preprocessing module is used to obtain target processing data batch;Loading module is used to load target processing data batch to deep interest network model;Update module is used to form new historical memory unit and update historical data processing memory field.The application uses improved Hopfield method to realize commercial data preprocessing and loading control, with the advantage of strong adaptability.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence data processing technology, and in particular to a data preprocessing and loading control system for commercial AI models. Background Technology

[0002] As commercial AI models are increasingly applied in product recommendation, user behavior analysis, marketing decision-making, and traffic distribution scenarios, commercial data typically undergoes processing such as missing data completion, anomaly handling, label correction, field alignment, and batch loading before entering the model. Current technologies often employ rule-based filtering, priority sorting, or hierarchical scheduling based on the static quality status of the current batch of data before inputting the processed data into the recommendation or ranking model to improve model input quality and system efficiency.

[0003] However, most existing technologies make preprocessing and loading decisions based solely on the current batch of business data, lacking the ability to leverage historically validated processing experience. This makes it difficult to integrate historical batch input states, preprocessing action chains, loading action chains, model performance results, and resource impact results for use in current batch control. Furthermore, existing technologies typically treat preprocessing and loading as independent processes, making it difficult to establish a joint trajectory reconstruction mechanism that considers the current batch's conflict characteristics, cache usage characteristics, loading queue characteristics, resource load characteristics, and model stage characteristics. This results in insufficient data processing path adaptability and poor model input stability and resource utilization coordination.

[0004] Therefore, how to provide a data preprocessing and loading control system for commercial AI models is a problem that urgently needs to be solved by those skilled in the art. Summary of the Invention

[0005] One objective of this invention is to propose a data preprocessing and loading control system for commercial AI models. This invention uses an improved Hopfield method to achieve commercial data preprocessing and loading control, which has the advantage of strong adaptability.

[0006] A data preprocessing and loading control system for a commercial AI model according to an embodiment of the present invention includes the following steps: The acquisition module is used to acquire the current batch of business data and obtain the corresponding data. The state extraction module is used to perform state extraction processing to obtain features and construct the current batch query state; The memory construction module is used to perform unified encoding processing, form multiple historical memory units, and construct a historical data processing memory field. The associative retrieval module is used to input the current batch query status into the improved Hopfield network and perform associative memory convergence retrieval processing to obtain a set of candidate memory units corresponding to the current batch query status. The trajectory reconstruction module is used to perform correlation analysis and combine the current batch conflict characteristics, current cache usage characteristics, current loading queue characteristics, current resource load characteristics and current model stage characteristics to perform trajectory reconstruction processing and generate target preprocessing action chain and target loading action chain; The preprocessing module is used to perform preprocessing on the current batch of business data sequentially according to the target preprocessing action chain to obtain the target batch of data to be processed. The loading module is used to load batches of target processing data into the deep interest network model; The update module is used to read the results, perform write-back processing, form new historical memory units, and update the historical data processing memory field.

[0007] Optionally, the corresponding data includes historical batch input status, historical preprocessing action chain, historical loading action chain, historical model benefit results, and historical resource impact results.

[0008] Optionally, constructing the current batch query status includes: Read the field content in the current batch of business data, and count the number of complete records, missing records, abnormal records, and fluctuating records according to the order of record entries to form the input characteristics of the current batch; Read the source identifiers from the current batch of business data and perform source classification processing to form the source characteristics of the current batch; Read the field content, business tag records, and timestamp records from the current batch of business data, perform conflict identification processing, obtain field conflict results, tag conflict results, and time conflict results, and form the conflict characteristics of the current batch; Read the timestamp records in the current batch of business data, calculate the time interval between the arrival time and processing time of each record, and form the timeliness characteristics of the current batch; Read the business tag records, batch identifier records, and business data in the field content of the current business data batch, perform value identification processing, obtain the record value result and batch value result, and form the value feature of the current batch; Read the cache usage record, loading queue record, resource call record and model running stage record corresponding to the current batch of business data entering the processing link, and form the current loading queue characteristics, current resource load characteristics and current model stage characteristics; The current batch input features, current batch source features, current batch conflict features, current batch timeliness features, current batch value features, current cache usage features, current loading queue features, current resource load features, and current model stage features are concatenated in a fixed order. The concatenated results are then subjected to unified dimensional mapping and sequential encoding to obtain the current batch query status.

[0009] Optionally, forming multiple historical memory units and constructing a historical data processing memory field includes: Read the historical batch input status, historical preprocessing action chain, historical loading action chain, historical model revenue results and historical resource impact results corresponding to each historical business data batch, and perform corresponding sorting and processing according to the batch affiliation of the historical business data batches; The historical batch input status, historical preprocessing action chain, historical loading action chain, historical model benefit result and historical resource impact result corresponding to the same historical business data batch are spliced ​​together in a fixed order to form a historical batch combination sequence. Perform a unified dimension mapping process on the historical batch combination sequence to form a historical batch mapping sequence; Perform sequential encoding processing on the historical batch mapping sequence to form a historical batch encoding sequence; Use the historical batch coding sequence as the historical memory unit for the corresponding historical business data batch; All historical memory units are processed by performing set construction according to the batch order of historical business data batches to obtain the historical data processing memory field.

[0010] Optionally, the generation of the candidate memory unit set includes: The current batch query status is written into the improved Hopfield network, and all historical memory units in the historical data processing memory field are written into the improved Hopfield network. The improved Hopfield network includes a query status input layer, a memory mapping layer, an association matching layer, a constraint convergence layer, and a candidate output layer. In the query status input layer, the query status of the current batch is arranged into a query status matrix according to the feature order; The query state matrix is ​​input into the memory mapping layer, and all historical memory units in the historical data processing memory field are arranged into a memory state matrix in batch order. The query state matrix and the memory state matrix are mapped in the memory mapping layer to obtain the query representation matrix and the memory representation matrix. In the associative matching layer, matching calculations are performed based on the query representation matrix and the memory representation matrix to obtain the memory matching matrix, and the memory matching matrix is ​​normalized to form the initial associative activation matrix; In the constrained convergence layer, the initial associative activation matrix is ​​used as the basis for network state updates, and iterative state update processing is performed in combination with the arrangement relationship of each historical memory unit in the historical data processing memory field to obtain the convergence state matrix. When the difference between two consecutive convergence energy results satisfies the convergence termination condition, the iterative state update is stopped, and the current convergence state matrix is ​​taken as the final associative convergence result. In the candidate output layer, a matching and sorting process is performed based on the final associative convergence result and each historical memory unit to obtain a set of candidate memory units.

[0011] Optionally, the generation of the target preprocessing action chain and the target loading action chain includes: Read the historical preprocessing action chain, historical loading action chain, historical model benefit result and historical resource impact result from the candidate memory unit set, and perform sorting and processing according to the correspondence of the same candidate memory unit; Perform correlation analysis on the historical preprocessing action chain, historical loading action chain, historical model benefit results and historical resource impact results corresponding to the same candidate memory unit to form a candidate processing trajectory; The current batch conflict characteristics, current cache usage characteristics, current loading queue characteristics, current resource load characteristics, and current model stage characteristics are loaded into the candidate processing trajectory to form a constraint processing trajectory; Perform trajectory reconstruction processing on the constraint processing trajectory to obtain the reconstruction preprocessing action chain and the reconstruction loading action chain; The reconstructed preprocessing action chain is determined as the target preprocessing action chain corresponding to the current business data batch, and the reconstructed loading action chain is determined as the target loading action chain corresponding to the current business data batch.

[0012] Optionally, the formation of the target processing data batch includes: Read the current batch of business data and the target preprocessing action chain, and determine the processing order according to the order of actions in the target preprocessing action chain; The current batch of commercial data is processed in the following order: missing data completion, anomaly compression, label correction, field alignment, conflict resolution, source weight reduction, redundancy trimming, and batch splitting. The trimmed data batch is split into multiple processing sub-batches according to the record ownership relationship, and the multiple processing sub-batches are combined to form the target processing data batch.

[0013] Optionally, loading the target processing data batches into the deep interest network model includes: Read the target processing data batch and the target loading action chain, and determine the loading order according to the order of actions in the target loading action chain; According to the loading order, priority injection processing, hierarchical injection processing, staged injection processing, post-cached injection processing, delayed injection processing, and batch splitting and alternating injection processing are performed on the target data batches to form the target loaded data batches; The target data is loaded into the deep interest network model in batches according to the loading order.

[0014] Optionally, the process of forming new historical memory units and updating the historical data processing memory field includes: Read the current model benefit results output by the deep interest network model for the target data batch, and read the current resource impact results corresponding to the target data batch loading process; The current batch query status, target preprocessing action chain, target loading action chain, current model profit result and current resource impact result are concatenated in a fixed order to form the current batch write-back sequence. Perform a uniform dimension mapping process on the current batch write-back sequence to form the current batch mapped write-back sequence; Perform sequential encoding processing on the current batch mapping write-back sequence to form the current batch encoded write-back sequence; Write the current batch code back into the sequence as a new historical memory unit; The new historical memory units are written into the historical data processing memory field, thus completing the update of the historical data processing memory field.

[0015] The beneficial effects of this invention are: This invention extracts the state of the current batch of business data and constructs a historical data processing memory field by combining historical batch input states, historical preprocessing action chains, historical loading action chains, historical model benefit results, and historical resource impact results. This allows the system to move beyond relying solely on the static state of the current batch to generate processing decisions. Instead, it leverages proven historical processing experience to generate target preprocessing and loading action chains for the current batch of business data. Compared to existing technologies, this invention improves the matching degree between the business data processing path and current data characteristics, resource status, and model stage, thereby enhancing the targeting of data preprocessing and loading control.

[0016] Meanwhile, this invention improves the Hopfield network to perform associative memory convergence retrieval and, based on the candidate memory unit set, combines current batch conflict characteristics, current cache usage characteristics, current loading queue characteristics, current resource load characteristics, and current model stage characteristics to perform trajectory reconstruction, enabling preprocessing and loading actions to form a unified collaborative trajectory. This not only improves the data quality and structural consistency of the input deep interest network model but also reduces resource fluctuations and processing mismatch risks during the loading process. Furthermore, this invention writes the current model gain results and current resource impact results back to the historical data processing memory field, enabling the system to continuously accumulate processing experience and dynamically optimize subsequent processing strategies. Attached Figure Description

[0017] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used in conjunction with embodiments of the invention to explain the invention and do not constitute a limitation thereof. In the drawings: Figure 1 This is an overall flowchart of a data preprocessing and loading control system for a commercial AI model proposed in this invention; Figure 2 This is a flowchart illustrating the formation of historical memory units and the construction of historical data processing memory fields in a data preprocessing and loading control system for a commercial AI model proposed in this invention. Figure 3 This is a flowchart of the candidate processing trajectory generation and target preprocessing action chain and target loading action chain reconstruction in a data preprocessing and loading control system for a commercial AI model proposed in this invention. Detailed Implementation

[0018] The present invention will now be described in further detail with reference to the accompanying drawings. These drawings are simplified schematic diagrams, illustrating only the basic structure of the invention, and therefore only show the components relevant to the invention.

[0019] refer to Figures 1-3 A data preprocessing and loading control system for commercial AI models includes the following steps: The acquisition module is used to acquire the current batch of business data, and to acquire the historical batch input status, historical preprocessing action chain, historical loading action chain, historical model benefit results, and historical resource impact results corresponding to the historical batches of business data. The state extraction module is used to perform state extraction processing on the current batch of business data to obtain the current batch input features, current batch source features, current batch conflict features, current batch timeliness features, current batch value features, current cache usage features, current loading queue features, current resource load features, and current model stage features, and construct the current batch query state; The memory construction module is used to perform unified encoding processing on historical batch input states, historical preprocessing action chains, historical loading action chains, historical model benefit results, and historical resource impact results, forming multiple historical memory units and constructing a historical data processing memory field. The associative retrieval module is used to input the current batch query status into the improved Hopfield network, and use the historical data processing memory field as the associative memory convergence object to perform associative memory convergence retrieval processing to obtain the set of candidate memory units corresponding to the current batch query status. The trajectory reconstruction module is used to perform correlation analysis on the historical preprocessing action chain, historical loading action chain, historical model benefit results and historical resource impact results in the candidate memory unit set, and to perform trajectory reconstruction processing in combination with the current batch conflict characteristics, current cache usage characteristics, current loading queue characteristics, current resource load characteristics and current model stage characteristics to generate the target preprocessing action chain and target loading action chain corresponding to the current business data batch. The preprocessing module is used to sequentially perform missing data completion, anomaly compression, label correction, field alignment, conflict resolution, source weight reduction, redundancy pruning, and batch splitting on the current batch of commercial data according to the target preprocessing action chain, so as to obtain the target data batch. The loading module is used to perform priority injection, hierarchical injection, staged injection, post-caching injection, delayed injection, and batch-alternating injection on batches of target processing data according to the target loading action chain, so as to load batches of target processing data into the deep interest network model. The update module is used to read the current model benefit results output by the deep interest network model for the target processing data batch, and read the current resource impact results corresponding to the target processing data batch loading process. It also performs write-back processing on the current batch query status, target preprocessing action chain, target loading action chain, current model benefit results, and current resource impact results to form a new historical memory unit and update the historical data processing memory field.

[0020] In this embodiment, the historical business data batches correspond to the historical batch input status, historical preprocessing action chain, historical loading action chain, historical model benefit results, and historical resource impact results.

[0021] In this implementation, constructing the current batch query status includes: Read the field content in the current batch of business data, and count the number of complete records, missing records, abnormal records, and fluctuating records according to the order of record entries to form the input characteristics of the current batch; Read the source identifiers from the current batch of business data, and perform source classification processing on each record according to the source identifiers. Count the number of records with consistent source, the number of records with conflicting source, and the number of records with offset source to form the source characteristics of the current batch; Read the field content, business tag record, and timestamp record in the current batch of business data, and perform conflict identification processing on each record based on the field content, business tag record, and timestamp record to obtain field conflict results, tag conflict results, and time conflict results. Based on the field conflict results, tag conflict results, and time conflict results, form the conflict characteristics of the current batch. Read the timestamp records in the current batch of business data, calculate the time interval between the arrival time and processing time of each record, and form the timeliness characteristics of the current batch based on the time interval; Read the business tag records, batch identifier records, and business data in the field content of the current business data batch, perform value identification processing, obtain the record value result and the batch value result, and form the current batch value feature based on the record value result and the batch value result; Read the cache usage record, loading queue record, resource call record and model running stage record corresponding to the current batch of business data entering the processing link, and form the current cache usage feature based on the cache usage record, the current loading queue feature based on the loading queue record, the current resource load feature based on the resource call record, and the current model stage feature based on the model running stage record; The current batch input features, current batch source features, current batch conflict features, current batch timeliness features, current batch value features, current cache usage features, current loading queue features, current resource load features, and current model stage features are concatenated in a fixed order. The concatenated results are then subjected to unified dimensional mapping and sequential encoding to obtain the current batch query status.

[0022] In this embodiment, forming multiple historical memory units and constructing a historical data processing memory field includes: Read the historical batch input status, historical preprocessing action chain, historical loading action chain, historical model revenue results and historical resource impact results corresponding to each historical business data batch, and perform corresponding sorting and processing according to the batch affiliation of the historical business data batches; The historical batch input status, historical preprocessing action chain, historical loading action chain, historical model benefit result and historical resource impact result corresponding to the same historical business data batch are spliced ​​together in a fixed order to form a historical batch combination sequence. The fixed order is historical batch input status, historical preprocessing action chain, historical loading action chain, historical model benefit result and historical resource impact result. Perform a unified dimension mapping process on the historical batch combination sequence to bring the state data, action data, profit data and impact data in the historical batch combination sequence to the same mapping scale, forming a historical batch mapping sequence; The unified dimensional mapping process is as follows: For continuous numerical items in the historical batch combination sequence, a linear interval mapping method is used to map each continuous numerical item to a value between zero and one; for action category items in the historical preprocessing action chain and historical loading action chain, a category number mapping method is used to map different action categories to different integer values; for the grade items in the historical model return results, an ordinal grade mapping method is used to map higher grades to larger ordinal values; for the grade items in the historical resource impact results, a reverse ordinal grade mapping method is used to map larger resource impacts to smaller ordinal values; and for vacant positions, fixed placeholder values ​​are written to ensure that data at each position in the historical batch mapping sequence can enter the same mapping scale. Perform sequential encoding processing on the historical batch mapping sequence so that the order of the historical batch mapping sequence is preserved in the encoding result, forming a historical batch encoding sequence; The sequential encoding process is as follows: Each mapping value is assigned a position number according to its position in the historical batch mapping sequence; the position numbers increment sequentially according to the order of appearance in the historical batch combination sequence; the original order is preserved for data positions belonging to historical batch input states, historical preprocessing action chains, historical loading action chains, historical model benefit results, and historical resource impact results; then, the position numbers are combined with the corresponding mapping values ​​to form a historical batch encoding sequence, ensuring that the same mapping value can be distinguished even when it appears in different positions. Use the historical batch coding sequence as the historical memory unit for the corresponding historical business data batch; All historical memory units are processed by performing set construction according to the batch order of historical business data batches to obtain the historical data processing memory field.

[0023] In this embodiment, the generation of the candidate memory unit set includes: The current batch query status is written into the improved Hopfield network, and all historical memory units in the historical data processing memory field are written into the improved Hopfield network. The improved Hopfield network includes a query status input layer, a memory mapping layer, an association matching layer, a constraint convergence layer, and a candidate output layer. In the query status input layer, the query status of the current batch is arranged into a query status matrix according to the feature order; The query state matrix is ​​input into the memory mapping layer, and all historical memory units in the historical data processing memory field are arranged into a memory state matrix in batch order. The query state matrix and the memory state matrix are mapped in the memory mapping layer to obtain the query representation matrix and the memory representation matrix. The mapping process refers to multiplying the query state matrix with the query mapping weight matrix and multiplying the memory state matrix with the memory mapping weight matrix. In the associative matching layer, matching calculations are performed based on the query representation matrix and the memory representation matrix to obtain the memory matching matrix, and the memory matching matrix is ​​normalized to form the initial associative activation matrix; In the constrained convergence layer, the initial associative activation matrix is ​​used as the basis for network state updates, and iterative state update processing is performed in combination with the arrangement relationship of each historical memory unit in the historical data processing memory field to obtain the convergence state matrix. The iterative state update process is as follows: using the initial association activation result as the current network state, multiplying the current network state bit by bit with the encoded values ​​corresponding to all historical memory units and accumulating them in the order of historical memory units to obtain the memory recall result; then performing weighted fusion of the memory recall result and the encoded value corresponding to the current batch query state according to the same position to obtain the update input result; performing saturation compression processing on the values ​​at each position in the update input result to keep the values ​​at each position within a fixed value range, and using the compressed result as the current network state for the next iteration; repeating the memory recall result calculation, update input result calculation, and saturation compression processing to form a network state sequence; When the difference between two consecutive convergence energy results satisfies the convergence termination condition, the iterative state update is stopped, and the current convergence state matrix is ​​taken as the final associative convergence result. The convergence energy result is as follows: For the current network state obtained in each iteration, first calculate the sum of matching between the current network state and all historical memory units, then calculate the sum of the changes in all position values ​​in the current network state, and subtract the sum of the changes in position values ​​from the sum of matching to obtain the convergence energy result; The convergence termination condition is as follows: the difference between two adjacent convergence energy results is less than a fixed energy threshold, and the fixed energy threshold is met for two consecutive times, or the number of iterations reaches a fixed iteration limit; In the candidate output layer, a matching and sorting process is performed based on the final associative convergence result and each historical memory unit to obtain a set of candidate memory units.

[0024] In this embodiment, the generation of the target preprocessing action chain and the target loading action chain includes: Read the historical preprocessing action chain, historical loading action chain, historical model benefit result and historical resource impact result from the candidate memory unit set, and perform sorting and processing according to the correspondence of the same candidate memory unit; Perform correlation analysis on the historical preprocessing action chain, historical loading action chain, historical model benefit results and historical resource impact results corresponding to the same candidate memory unit to form a candidate processing trajectory; The specific steps for forming a candidate processing trajectory are as follows: according to the original sequence of the historical preprocessing action chain and the historical loading action chain in the same candidate memory unit, each preprocessing action in the historical preprocessing action chain and each loading action in the historical loading action chain are sequentially connected; then the historical model benefit result is written to the end of the corresponding candidate memory unit, and the historical resource impact result is written to the position after the historical model benefit result, so that the same candidate memory unit corresponds to a unique candidate processing trajectory. The current batch conflict characteristics, current cache usage characteristics, current loading queue characteristics, current resource load characteristics, and current model stage characteristics are loaded into the candidate processing trajectory to form a constraint processing trajectory; The specific process of forming a constraint processing trajectory is as follows: the conflict features of the current batch are written into the position set corresponding to the historical preprocessing action chain in the candidate processing trajectory, and the current cache occupancy features, current loading queue features, current resource load features, and current model stage features are written into the position set corresponding to the historical loading action chain in the candidate processing trajectory. The writing method is position-bound writing, that is, each feature is only written to its corresponding action position, without changing the original action order of the candidate processing trajectory. Perform trajectory reconstruction processing on the constraint processing trajectory, adjust the historical preprocessing action chain in the candidate processing trajectory based on the current batch conflict characteristics to obtain the reconstructed preprocessing action chain, and adjust the historical loading action chain in the candidate processing trajectory based on the current cache occupancy characteristics, current loading queue characteristics, current resource load characteristics, and current model stage characteristics to obtain the reconstructed loading action chain; The specific steps for adjusting the historical preprocessing action chain in the candidate processing trajectory based on the current batch conflict characteristics are as follows: When the number of conflicts corresponding to field conflict results in the current batch conflict characteristics is greater than the number of conflicts corresponding to label conflict results, the field alignment processing is moved forward to before the label correction processing; when the number of conflicts corresponding to time conflict results is greater than the number of conflicts corresponding to field conflict results, the conflict resolution processing is moved forward to before the field alignment processing; when the number of conflicts corresponding to source conflict results is greater than the number of missing records, the source deweighting processing is moved forward to before the redundancy pruning processing. The historical loading action chain in the candidate processing trajectory is adjusted based on the current cache usage characteristics, current loading queue characteristics, current resource load characteristics, and current model stage characteristics as follows: When the cache usage value corresponding to the current cache usage characteristic is greater than the cache threshold, the post-cached injection processing is moved forward; when the loading queue length corresponding to the current loading queue characteristic is greater than the queue threshold, the delayed injection processing is moved forward; when the resource load value corresponding to the current resource load characteristic is greater than the load threshold, the batch-alternating injection processing is moved forward; when the current model stage characteristic corresponds to the training stage, the hierarchical injection processing is moved forward; when the current model stage characteristic corresponds to the inference stage, the priority injection processing is moved forward. The reconstructed preprocessing action chain is determined as the target preprocessing action chain corresponding to the current business data batch, and the reconstructed loading action chain is determined as the target loading action chain corresponding to the current business data batch.

[0025] In this embodiment, the formation of the target processing data batch includes: Read the current batch of business data and the target preprocessing action chain, and determine the processing order according to the order of actions in the target preprocessing action chain; The current batch of commercial data is processed in the following order: missing data completion, anomaly compression, label correction, field alignment, conflict resolution, source weight reduction, redundancy trimming, and batch splitting. The trimmed data batch is split into multiple processing sub-batches according to the record ownership relationship, and the multiple processing sub-batches are combined to form the target processing data batch. The missing data completion process is as follows: for missing fields in the current batch of business data, first read the non-missing field values ​​in the same record to complete the same record; when the same record cannot be completed, read the historical valid values ​​in the same batch that are in the same position as the missing field to complete the batch; after the completion is completed, a completed data batch is generated. The specific abnormal compression process is as follows: perform boundary compression on abnormal values ​​that exceed the valid value range of the field in the data batch, retain single labels for repeated abnormal labels, and truncate the fluctuation amplitude for continuous abnormal fluctuation records to generate compressed data batches. The batch splitting process is as follows: based on the source attribution and business tag attribution in the record attribution relationship, the cropped data batch is divided into multiple processing sub-batches, and records with the same source attribution and the same business tag attribution are kept in the same processing sub-batch.

[0026] In this embodiment, loading the target processing data batches into the deep interest network model includes: Read the target processing data batch and the target loading action chain, and determine the loading order according to the order of actions in the target loading action chain; According to the loading order, priority injection processing, hierarchical injection processing, staged injection processing, post-cached injection processing, delayed injection processing, and batch splitting and alternating injection processing are performed on the target data batches to form the target loaded data batches; The priority injection process is as follows: according to the priority order of each batch of fragments in the target loading action chain, the batch fragments with higher priority are arranged first, and the batch fragments with lower priority are arranged last. The layered injection process is as follows: according to the hierarchical order of each batch of fragments in the target loading action chain, the batch fragments at the same level are grouped into the same level and arranged from front to back according to the hierarchy; The phased injection process specifically involves grouping batches of segments belonging to the same phase into the same phase according to the phase order of each batch of segments in the target loading action chain. The specific process of post-caching injection is as follows: first write the batch fragments that need to be cached into the cache area, and then output them in the order of cache release; The delayed injection process specifically involves adjusting batch segments with later delay positions to subsequent time positions. The batch-alternating injection process specifically involves splitting different batches of segments and arranging them in a predetermined alternating order to form the target loading data batch. The target data is loaded into the deep interest network model in batches according to the loading order.

[0027] In this embodiment, forming new historical memory units and updating the historical data processing memory field includes: Read the current model benefit results output by the deep interest network model for the target data batch, and read the current resource impact results corresponding to the target data batch loading process; The specific benefits of the current model are as follows: at least one of the following is read from the output of the deep interest network model: the improvement in click-through rate prediction, the improvement in conversion rate prediction, and the improvement in ranking position. The current resource impact results are specifically: at least one of the following values ​​is read from the resource monitoring results of the target data batch loading process: cache usage change value, loading queue change value, and resource load change value; The current batch query status, target preprocessing action chain, target loading action chain, current model benefit result, and current resource impact result are concatenated in a fixed order to form the current batch write-back sequence. The fixed order is: current batch query status, target preprocessing action chain, target loading action chain, current model benefit result, and current resource impact result. Perform a unified dimension mapping process on the current batch of write-back sequences to bring the state data, action data, profit data and impact data in the current batch of write-back sequences to the same mapping scale, forming the current batch of mapped write-back sequences; The unified dimension mapping process is as follows: the query status of the current batch in the current batch write-back sequence is processed by a combination of continuous numerical mapping and category number mapping; the target preprocessing action chain and the target loading action chain are processed by action category number mapping; the current model benefit result is processed by sequential level mapping; and the current resource impact result is processed by reverse sequential level mapping, so that all data in the current batch mapping write-back sequence are at the same mapping scale. Perform sequential encoding on the current batch mapping write-back sequence to preserve the order of the current batch mapping write-back sequence in the encoding result, forming the current batch encoded write-back sequence; Write the current batch code back into the sequence as a new historical memory unit; Write the new historical memory unit into the historical data processing memory field to complete the update of the historical data processing memory field; The specific steps to update the historical data processing memory field are as follows: new historical memory units are appended to the end of the historical data processing memory field according to the writing order corresponding to the current business data batch, and the batch order index in the historical data processing memory field is updated simultaneously, so that the new historical memory units can be read in order during subsequent associative memory convergence retrieval processing.

[0028] Example 1: To verify the feasibility of this invention in practice, it was applied to the product recommendation business environment of a large e-commerce company. This company's recommendation service provides data support for multiple business entry points, including mobile homepage recommendations, post-search recommendations, and activity page recommendations. The backend continuously receives user click logs, exposure logs, add-to-cart logs, transaction logs, product attribute data, store attribute data, marketing tag data, inventory status data, and model operation monitoring data. Since these data come from the transaction system, content system, user behavior system, tag system, and resource scheduling system, the data writing times between different sources are inconsistent, and the field formats are not entirely uniform. The tag names, update times, and weight identifiers for the same product often differ across different sources.

[0029] When the recommendation pipeline enters promotional periods or peak traffic periods, the number of newly arriving commercial data batches increases significantly. This also leads to a rise in missing fields, abnormal labels, source conflicts, and time misalignments in the data. This makes the traditional approach of cleaning data based solely on the current batch and then feeding it into the deep interest network model in a fixed order unsustainable in real-world environments. On one hand, the processing pipeline easily mixes high-value, time-sensitive data with data from low-reliability sources in the same round, resulting in an unreasonable preprocessing order. On the other hand, the loading pipeline often cannot dynamically adjust the injection method based on cache usage, loading queue congestion, and changes in model stage, further impacting the recommendation model's absorption of input data.

[0030] This invention is deployed and validated in such a business environment, focusing on solving the problem of the lack of linkage between newly arriving batches of business data and historically validated effective processing experience. In specific applications, the acquisition module continuously receives the current batch of business data and simultaneously retrieves the historical batch input status, historical preprocessing action chain, historical loading action chain, historical model benefit results, and historical resource impact results corresponding to the historical batches of business data. Subsequently, the state extraction module organizes and identifies the current batch of business data, forming the current batch input features, current batch source features, current batch conflict features, current batch timeliness features, current batch value features, current cache usage features, current loading queue features, current resource load features, and current model stage features, and further constructs the current batch query state; the memory construction module concatenates and uniformly encodes the historical batch input states, historical preprocessing action chains, historical loading action chains, historical model benefit results, and historical resource impact results in a fixed order, forming historical memory units, and then constructs a historical data processing memory field; the associative retrieval module inputs the current batch query state into the improved Hopfield network, enabling the current batch query state to perform associative memory convergence retrieval with multiple historical memory units in the historical data processing memory field, thereby obtaining a set of candidate memory units.

[0031] The trajectory reconstruction module forms candidate processing trajectories based on the candidate memory unit set, and generates target preprocessing action chains and target loading action chains by combining the current batch conflict characteristics, current cache usage characteristics, current loading queue characteristics, current resource load characteristics, and current model stage characteristics. The preprocessing module sequentially executes missing data completion, anomaly compression, label correction, field alignment, conflict resolution, source weight reduction, redundancy pruning, and batch splitting according to the target preprocessing action chain to obtain the target processing data batch. The loading module then executes priority injection, hierarchical injection, stage-based injection, post-cached injection, delayed injection, and alternating batch injection according to the target loading action chain, and loads the processed target loading data batch into the deep interest network model. Finally, the system reads the current model benefit result output by the deep interest network model and the current resource impact result corresponding to the loading process, writes the current batch query status, target preprocessing action chain, target loading action chain, current model benefit result, and current resource impact result back to form new historical memory units and update the historical data processing memory field.

[0032] Continuous operation demonstrates that this invention can more stably complete the construction of the current batch query state, retrieval of candidate memory units, generation of target preprocessing action chains, and generation of target loading action chains when faced with commercial data that has many source conflicts, inconsistent field updates, and significant differences in batch timeliness. It also continuously supplements new and effective experience, thereby constantly improving the historical data processing memory field. This effectively solves the problems in the prior art, such as the inability of historical experience to participate in the current batch processing decision, the separation between preprocessing and loading, and the insufficient stability of model input.

[0033] Table 1. Overall Performance Comparison of History Memory-Driven Data Preprocessing and Loading Control

[0034] As shown in Table 1, the method of this invention achieves optimal results across all four retention evaluation dimensions. Compared to the traditional rule-based cleaning and fixed-order loading method, the data utilization rate of this invention increases from 78.6% to 91.8%. This indicates that this invention does not simply increase the cleaning intensity, but rather prioritizes the use of historical memory units and historical data processing memory fields to call upon processing experience that matches the current batch, thus effectively retaining more data that would otherwise be coarsely filtered or inefficiently processed and entering subsequent links. Correspondingly, the recommended click rate of this invention increases by 11.7%, indicating that the data quality entering the deep interest network model is more consistent, and the loading rhythm is better matched with the model's absorption method.

[0035] From the perspective of resource-side indicators, the resource impact index of this invention is 0.47, which is lower than 0.82 of the traditional method, 0.69 of the static scoring preprocessing plus priority loading method, and 0.61 of the newly added comparison method based on historical memory retrieval processing chain selection method. This data shows that this invention not only focuses on whether data can enter the model, but also on whether the entry method will cause fluctuations in cache, queue and resource load. The reason is that after the candidate processing trajectory is generated, this invention does not directly reuse the historical action chain, but further combines the current cache occupancy characteristics, current loading queue characteristics, current resource load characteristics and current model stage characteristics to reconstruct the target loading action chain. Therefore, it can arrange actions such as post-cache injection, delayed injection, and batch-alternating injection in more appropriate positions, thereby reducing instantaneous resource impact.

[0036] The newly added comparative method, a processing chain selection method based on historical memory retrieval, improves data utilization from 84.9% to 88.7% and the recommendation click improvement rate from 5.4% to 8.1% compared to the static scoring preprocessing plus priority loading method. This demonstrates that incorporating historical processing experience itself brings significant benefits. However, further comparison with this invention shows that it still improves data utilization by 3.1 percentage points, recommendation click improvement rate by 3.6 percentage points, reduces resource impact index by 0.14, and improves loading stability score by 4.7 points. This indicates that simply performing historical memory retrieval is insufficient; the key improvement comes from subsequent trajectory reconstruction processing. This involves adjusting the target preprocessing action chain based on the current batch conflict characteristics, and then adjusting the target loading action chain based on the current cache usage characteristics, current loading queue characteristics, current resource load characteristics, and current model stage characteristics, making historical experience not only referable but also reconstructable.

[0037] In terms of loading stability score, this invention achieved 90.6 points, which directly reflects the continuity and orderliness of the data injection process. Traditional methods often treat preprocessing and loading as two separate steps, while this invention places the target preprocessing action chain and the target loading action chain in the same candidate processing trajectory for joint generation. Therefore, after preprocessing, different batches of segments can be written into the deep interest network model at a smoother rhythm, reducing fluctuations caused by sudden congestion, batch stacking, and uneven injection.

[0038] Based on Table 1, it can be concluded that the improvement of this invention is not an accidental improvement on a single indicator, but the result of simultaneous optimization in four aspects: data utilization, model benefits, resource impact, and loading stability. This better reflects its overall advantages in the scenario of controlling the data entry point of commercial AI models.

[0039] The above are merely preferred embodiments of the present invention, but the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in the present invention, based on the technical solution and inventive concept of the present invention, should be covered within the scope of protection of the present invention.

Claims

1. A data preprocessing and loading control system for a commercial AI model, characterized in that, Includes the following steps: The acquisition module is used to acquire the current batch of business data and obtain the corresponding data. The state extraction module is used to perform state extraction processing to obtain features and construct the current batch query state; The memory construction module is used to perform unified encoding processing, form multiple historical memory units, and construct a historical data processing memory field. The associative retrieval module is used to input the current batch query status into the improved Hopfield network and perform associative memory convergence retrieval processing to obtain a set of candidate memory units corresponding to the current batch query status. The trajectory reconstruction module is used to perform correlation analysis and combine the current batch conflict characteristics, current cache usage characteristics, current loading queue characteristics, current resource load characteristics and current model stage characteristics to perform trajectory reconstruction processing and generate target preprocessing action chain and target loading action chain; The preprocessing module is used to perform preprocessing on the current batch of business data sequentially according to the target preprocessing action chain to obtain the target batch of data to be processed. The loading module is used to load batches of target processing data into the deep interest network model; The update module is used to read the results, perform write-back processing, form new historical memory units, and update the historical data processing memory field.

2. The data preprocessing and loading control system for a commercial AI model according to claim 1, characterized in that, The corresponding data includes historical batch input status, historical preprocessing action chain, historical loading action chain, historical model benefit results, and historical resource impact results.

3. The data preprocessing and loading control system for a commercial AI model according to claim 1, characterized in that, The process of constructing the current batch query status includes: Read the field content in the current batch of business data, and count the number of complete records, missing records, abnormal records, and fluctuating records according to the order of record entries to form the input characteristics of the current batch; Read the source identifiers from the current batch of business data and perform source classification processing to form the source characteristics of the current batch; Read the field content, business tag records, and timestamp records from the current batch of business data, perform conflict identification processing, obtain field conflict results, tag conflict results, and time conflict results, and form the conflict characteristics of the current batch; Read the timestamp records in the current batch of business data, calculate the time interval between the arrival time and processing time of each record, and form the timeliness characteristics of the current batch; Read the business tag records, batch identifier records, and business data in the field content of the current business data batch, perform value identification processing, obtain the record value result and batch value result, and form the value feature of the current batch; Read the cache usage record, loading queue record, resource call record and model running stage record corresponding to the current batch of business data entering the processing link, and form the current loading queue characteristics, current resource load characteristics and current model stage characteristics; The current batch input features, current batch source features, current batch conflict features, current batch timeliness features, current batch value features, current cache usage features, current loading queue features, current resource load features, and current model stage features are concatenated in a fixed order. The concatenated results are then subjected to unified dimensional mapping and sequential encoding to obtain the current batch query status.

4. The data preprocessing and loading control system for a commercial AI model according to claim 1, characterized in that, The process of forming multiple historical memory units and constructing a historical data processing memory field includes: Read the historical batch input status, historical preprocessing action chain, historical loading action chain, historical model revenue results and historical resource impact results corresponding to each historical business data batch, and perform corresponding sorting and processing according to the batch affiliation of the historical business data batches; The historical batch input status, historical preprocessing action chain, historical loading action chain, historical model benefit result and historical resource impact result corresponding to the same historical business data batch are spliced ​​together in a fixed order to form a historical batch combination sequence. Perform a unified dimension mapping process on the historical batch combination sequence to form a historical batch mapping sequence; Perform sequential encoding processing on the historical batch mapping sequence to form a historical batch encoding sequence; Use the historical batch coding sequence as the historical memory unit for the corresponding historical business data batch; All historical memory units are processed by performing set construction according to the batch order of historical business data batches to obtain the historical data processing memory field.

5. The data preprocessing and loading control system for a commercial AI model according to claim 1, characterized in that, The generation of the candidate memory unit set includes: The current batch query status is written into the improved Hopfield network, and all historical memory units in the historical data processing memory field are written into the improved Hopfield network. The improved Hopfield network includes a query status input layer, a memory mapping layer, an association matching layer, a constraint convergence layer, and a candidate output layer. In the query status input layer, the query status of the current batch is arranged into a query status matrix according to the feature order; The query state matrix is ​​input into the memory mapping layer, and all historical memory units in the historical data processing memory field are arranged into a memory state matrix in batch order. The query state matrix and the memory state matrix are mapped in the memory mapping layer to obtain the query representation matrix and the memory representation matrix. In the associative matching layer, matching calculations are performed based on the query representation matrix and the memory representation matrix to obtain the memory matching matrix, and the memory matching matrix is ​​normalized to form the initial associative activation matrix; In the constrained convergence layer, the initial associative activation matrix is ​​used as the basis for network state updates, and iterative state update processing is performed in combination with the arrangement relationship of each historical memory unit in the historical data processing memory field to obtain the convergence state matrix. When the difference between two consecutive convergence energy results satisfies the convergence termination condition, the iterative state update is stopped, and the current convergence state matrix is ​​taken as the final associative convergence result. In the candidate output layer, a matching and sorting process is performed based on the final associative convergence result and each historical memory unit to obtain a set of candidate memory units.

6. The data preprocessing and loading control system for a commercial AI model according to claim 1, characterized in that, The generation of the target preprocessing action chain and the target loading action chain includes: Read the historical preprocessing action chain, historical loading action chain, historical model benefit result and historical resource impact result from the candidate memory unit set, and perform sorting and processing according to the correspondence of the same candidate memory unit; Perform correlation analysis on the historical preprocessing action chain, historical loading action chain, historical model benefit results and historical resource impact results corresponding to the same candidate memory unit to form a candidate processing trajectory; The current batch conflict characteristics, current cache usage characteristics, current loading queue characteristics, current resource load characteristics, and current model stage characteristics are loaded into the candidate processing trajectory to form a constraint processing trajectory; Perform trajectory reconstruction processing on the constraint processing trajectory to obtain the reconstruction preprocessing action chain and the reconstruction loading action chain; The reconstructed preprocessing action chain is determined as the target preprocessing action chain corresponding to the current business data batch, and the reconstructed loading action chain is determined as the target loading action chain corresponding to the current business data batch.

7. The data preprocessing and loading control system for a commercial AI model according to claim 1, characterized in that, The formation of the target processing data batch includes: Read the current batch of business data and the target preprocessing action chain, and determine the processing order according to the order of actions in the target preprocessing action chain; The current batch of commercial data is processed in the following order: missing data completion, anomaly compression, label correction, field alignment, conflict resolution, source weight reduction, redundancy trimming, and batch splitting. The trimmed data batch is split into multiple processing sub-batches according to the record ownership relationship, and the multiple processing sub-batches are combined to form the target processing data batch.

8. The data preprocessing and loading control system for a commercial AI model according to claim 1, characterized in that, The step of loading the target processing data batches into the deep interest network model includes: Read the target processing data batch and the target loading action chain, and determine the loading order according to the order of actions in the target loading action chain; According to the loading order, priority injection processing, hierarchical injection processing, staged injection processing, post-cached injection processing, delayed injection processing, and batch splitting and alternating injection processing are performed on the target data batches to form the target loaded data batches; The target data is loaded into the deep interest network model in batches according to the loading order.

9. A data preprocessing and loading control system for a commercial AI model according to claim 1, characterized in that, The process of forming new historical memory units and updating the historical data processing memory field includes: Read the current model benefit results output by the deep interest network model for the target data batch, and read the current resource impact results corresponding to the target data batch loading process; The current batch query status, target preprocessing action chain, target loading action chain, current model profit result and current resource impact result are concatenated in a fixed order to form the current batch write-back sequence. Perform a uniform dimension mapping process on the current batch write-back sequence to form the current batch mapped write-back sequence; Perform sequential encoding processing on the current batch mapping write-back sequence to form the current batch encoded write-back sequence; Write the current batch code back into the sequence as a new historical memory unit; The new historical memory units are written into the historical data processing memory field, thus completing the update of the historical data processing memory field.

Images