End-to-end data dynamic coordination system and method based on multi-modal data chain

Abstract

The application discloses an end-to-end data dynamic cooperation system and method based on a multi-modal data chain, and particularly relates to the technical field of data dynamic cooperation. Through a confidence resonance out-of-control coefficient, the error consistency risk in the system is accurately revealed. In combination with a multi-modal cooperation path shock coefficient constructed in each round of feedback optimization process, the convergence state and parameter disturbance trend of the system in the end-to-end feedback closed loop are quantified, and together with the confidence resonance out-of-control coefficient, an advertisement strategy cooperation / feedback double shock situation analysis model is formed. Through real-time output of an advertisement strategy cooperation / feedback double shock situation analysis index, accurate quantitative evaluation and risk judgment of the running state of a complex dynamic system are realized. Based on the advertisement strategy cooperation / feedback double shock situation analysis index, a modal confidence degree gating mechanism and an advertisement strategy feedback damping mechanism are constructed, and the system state is dynamically reconstructed from two key aspects of a cooperation signal entrance and a feedback path.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data dynamic collaboration, and more particularly, to an end-to-end data dynamic collaboration system and method based on a multi-modal data chain. BACKGROUND

[0002] With the continuous evolution of intelligent advertising delivery technology, how to realize the fusion analysis and accurate response of multi-modal information in multiple dimensions such as user behavior modeling, content understanding, and scene perception has become the core direction of digital advertising system optimization strategy. The current mainstream advertising platform widely accesses multi-modal data sources from user image interest, voice interaction, text browsing, geographic location, click behavior, social relationship, etc., forming a multi-modal data chain throughout the user's entire life cycle. The end-to-end dynamic collaboration optimization model based on this data chain can realize the linkage regulation of key strategies such as advertising content selection, display timing control, and user response prediction, significantly improving the relevance and conversion efficiency of advertising delivery.

[0003] However, the existing multi-modal collaboration method still faces many structural challenges in the actual operation of advertising strategy optimization, especially in the aspects of uneven modality quality, insufficient feedback adjustment strategy, and weak system dynamic convergence ability. Among them, one of the most representative technical problems is the coupling problem of confidence resonance out of control (collaboration drift) and multi-modal feedback path oscillation. Specifically, in the multi-modal collaboration process, low-quality modalities are easily misjudged and drifted by high-confidence modalities, forming a judgment result that is consistent on the surface but incorrect in essence. Once this "false consistency" is adopted by the feedback mechanism, it will further induce the repeated reinforcement of the system on the weight of the error modality, and then lead the whole system into a closed loop chain of feedback misadjustment-drift aggravation-convergence loss of control, forming a double oscillation situation of collaborative reasoning and feedback optimization.

[0004] Such phenomena not only destroy the original modality confidence system, but also cause high-quality modality information to be marginalized, shielded, or down-weighted in feedback, thereby causing cognitive bias and response errors of the system to the true state, and ultimately forming an unresolvable collaborative imbalance and state drift risk. SUMMARY

[0005] In order to overcome the above-mentioned defects of the prior art, the embodiments of the present application provide an end-to-end data dynamic collaboration system and method based on a multi-modal data chain to solve the problems raised in the background art.

[0006] To achieve the above-mentioned purpose, the present application provides the following technical solutions:

[0007] The end-to-end data dynamic collaboration method based on a multi-modal data chain comprises the following steps:

[0008] Step S1, obtain the confidence resonance runaway information of each modality in the response prediction process of the launch, and obtain the confidence resonance runaway coefficient according to the confidence resonance runaway information;

[0009] Step S2, obtain the collaborative path oscillation information between the multi-modal in each round of recommendation feedback update of the advertising strategy engine, and obtain the multi-modal collaborative path oscillation coefficient according to the collaborative path oscillation information;

[0010] Step S3, construct an advertising strategy collaborative / feedback double oscillation situation analysis model according to the confidence resonance runaway coefficient and the multi-modal collaborative path oscillation coefficient, obtain an advertising strategy collaborative / feedback double oscillation situation analysis index, and evaluate the unresolvable collaborative imbalance and state drift risk of the current collaborative system;

[0011] Step S4, construct an advertising modality confidence gating mechanism and an advertising strategy feedback damping mechanism to reconstruct the system state according to the unresolvable collaborative imbalance and state drift risk of the collaborative system.

[0012] In a preferred embodiment, by obtaining the confidence resonance runaway information of each modality in the response prediction process of the launch, the collaborative drift of the low-confidence modality caused by the dynamic guidance of the high-confidence modality is analyzed, and the confidence resonance runaway coefficient is obtained to measure the degree of collaborative drift of the low-confidence modality caused by the dynamic guidance of the high-confidence modality;

[0013] The logic of obtaining the confidence resonance runaway coefficient is as follows:

[0014] Obtain each modality The modality state at time t includes a semantic embedding vector And a confidence output Mark the modality state as ;

[0015] Calculate the confidence disturbance response value of the modality in the time interval ; ;

[0016] Define the modality disturbance graph All modalities constitute a node set of the modality disturbance graph , where is the number of modalities, and if modality , modality satisfy the following conditions simultaneously: , a directed edge is established;

[0017] Construct a disturbance flow matrix on the modality disturbance graph, which is used to represent whether there is a disturbance of "guiding low-confidence modalities by high-confidence modalities", where the elements of the disturbance flow matrix are: wherein is a perturbation state value, is a preset confidence threshold value;

[0018] for a low-confidence modality , the maximum length of its perturbed chain is obtained;

[0019] A perturbation propagation factor matrix is constructed according to the confidence perturbation response value, wherein the element of the perturbation propagation factor matrix is: wherein is a perturbation propagation factor, is a confidence perturbation response value of a modality , is a path hop count, indicating that the perturbation propagates from the modality to the modality in the modality perturbation graph; A resonance closed loop path set existing in the modality perturbation graph is identified

[0020] , and the resonance strength is calculated: ;

[0021] A confidence resonance runaway coefficient is calculated: wherein is the average path length of the perturbed chain of each low-confidence modality of the resonance closed loop path: , is the number of modes of the resonance closed loop path.

[0022] In a preferred embodiment, by obtaining the collaborative path shock information between the multi-modalities in each round of recommendation feedback update of the advertising strategy engine, the risk of continuous shock and non-convergence of the modality collaboration parameters in the closed loop due to the instability of the loss function or the over-regulation of the feedback in the end-to-end feedback adjustment is analyzed, and a multi-modal collaborative path shock coefficient is obtained to measure the risk degree of continuous shock and non-convergence of the modality collaboration parameters in the closed loop;

[0023] The logic of obtaining the multi-modal collaborative path shock coefficient is as follows:

[0024] For a modality , the collaborative parameter trajectory in the continuous feedback round is obtained, which represents the collaborative fusion parameter vector of the modality after the nth feedback;

[0025] A continuous perturbation response difference vector is calculated: , ,​ total feedback rounds for record;

[0026] calculate each feedback to modal according to continuous disturbance response difference vector co-integration parameter disturbance intensity rate of change

[0027] feedback to modal continuous disturbance response sequence Fourier transform: , calculate frequency domain disturbance energy spectrum , where high frequency sub-interval

[0028] analysis of non-convergent trend and "periodic oscillation instability" of modal in feedback to obtain local oscillation index of each modal

[0029] record oscillation synchronization index between modes for any modal pair

[0030] calculate multi-modal cooperative path oscillation coefficient according to local oscillation index and oscillation synchronization index , where modal number

[0031] In a preferred embodiment, an advertisement strategy synergy / feedback dual oscillation situation analysis model is constructed according to the confidence resonance runaway coefficient and the multi-modal cooperative path oscillation coefficient, and an advertisement strategy synergy / feedback dual oscillation situation analysis index is obtained, and the formula of the advertisement strategy synergy / feedback dual oscillation situation analysis model is as follows , where is the advertisement strategy synergy / feedback dual oscillation situation analysis index, is the confidence resonance runaway coefficient, is the multi-modal cooperative path oscillation coefficient, respectively represent the preset proportion coefficient of the confidence resonance runaway coefficient and the multi-modal cooperative path oscillation coefficient, and are both greater than 0.

[0032] In a preferred embodiment, the advertisement strategy synergy / feedback dual oscillation situation analysis index is compared with a preset advertisement strategy synergy / feedback dual oscillation situation analysis index threshold value to evaluate the risk of unresolvable synergy imbalance and state drift of the current synergy system, and the specific process is as follows:

[0033] ​​​​​​​​If the advertisement strategy synergy / feedback double shock situation analysis index is greater than the advertisement strategy synergy / feedback double shock situation analysis index threshold, a high synergy imbalance and state drift risk is generated;

[0034] If the advertisement strategy synergy / feedback double shock situation analysis index is less than or equal to the advertisement strategy synergy / feedback double shock situation analysis index threshold, a low synergy imbalance and state drift risk is generated.

[0035] In a preferred embodiment, when a high synergy imbalance and state drift risk is generated, an advertisement modality confidence gating mechanism and an advertisement strategy feedback damping mechanism are constructed according to the current confidence resonance out-of-control coefficient and the multi-modal synergy path shock coefficient to reconstruct the system state;

[0036] The advertisement modality confidence gating mechanism is as follows: , wherein is the confidence output of the modality , is the reconstructed confidence output.

[0037] The advertisement strategy feedback damping mechanism is as follows: , wherein is the Nth round of synergy fusion parameter vector of the modality , is the reconstructed (N+1)th round of synergy fusion parameter vector.

[0038] In a preferred embodiment, the end-to-end data dynamic synergy system based on a multi-modal data chain includes a resonance out-of-control module, a synergy path shock module, a risk comprehensive analysis module, and a state reconstruction module.

[0039] The resonance out-of-control module is configured to obtain confidence resonance out-of-control information of each modality in the prediction process of the response to the delivery, and obtain a confidence resonance out-of-control coefficient according to the confidence resonance out-of-control information.

[0040] The synergy path shock module is configured to obtain synergy path shock information between the multi-modal in each round of recommendation feedback update of the advertisement strategy engine, and obtain a multi-modal synergy path shock coefficient according to the synergy path shock information.

[0041] The risk comprehensive analysis module is configured to construct an advertisement strategy synergy / feedback double shock situation analysis model according to the confidence resonance out-of-control coefficient and the multi-modal synergy path shock coefficient, obtain an advertisement strategy synergy / feedback double shock situation analysis index, and evaluate the synergy imbalance and state drift risk of the current synergy system that cannot be recovered.

[0042] ​​A state reconstruction module is configured to reconstruct the system state according to the cooperative imbalance and state drift risk that cannot be recovered by the cooperative system, and build an advertisement modal confidence gating mechanism and an advertisement strategy feedback damping mechanism.

[0043] Technical effects and advantages of the present application:

[0044] 1、The present application constructs a confidence resonance loss-of-control coefficient, dynamically captures the phenomenon of "irrational cooperative drift" of low-confidence modes being dragged by high-confidence modes in system cooperative reasoning, and accurately reveals the "error consistency" risk in the system. Compared with the traditional static modal fusion method, this method is based on the joint modeling of modal confidence gradient and dynamic guidance effect, has stronger real-time perception ability and modal deviation sensitivity, can identify potential cooperative imbalance signs in advance before the system makes a global misjudgment, and further constructs a multi-modal cooperative path shock coefficient in each feedback optimization process to quantify the convergence state and parameter disturbance trend of the system in the end-to-end feedback loop, discover feedback shock behavior caused by misjudgment feedback, unstable loss function or out-of-control optimization step in time, and form an advertisement strategy cooperative / feedback double shock situation analysis model with the confidence resonance loss-of-control coefficient. Through the real-time output of the advertisement strategy cooperative / feedback double shock situation analysis index, the running state of the complex dynamic system is accurately quantitatively evaluated and judged, and based on the cooperative / feedback double shock situation analysis index, a modal confidence gating mechanism and a parameter feedback damping mechanism are further constructed to dynamically reconstruct the system state from the cooperative signal entrance and the feedback path. BRIEF DESCRIPTION OF DRAWINGS

[0045] In order to facilitate the understanding of those skilled in the art, the present application will be further described below with reference to the accompanying drawings;

[0046] Figure 1 The flowchart of the method of the present application embodiment 1 is shown in Figure 1.

[0047] Figure 2 The flowchart of the system of the present application embodiment 2 is shown in Figure 2. DETAILED DESCRIPTION

[0048] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor are within the scope of protection of the present application.

[0049] Embodiment 1: Figure 1 The end-to-end data dynamic cooperative method based on the multi-modal data chain of the present application is shown in Figure 1, which comprises the following steps:

[0050] Step S1, the confidence resonance runaway information of each mode in the response prediction process of putting is obtained, and the confidence resonance runaway coefficient is obtained according to the confidence resonance runaway information;

[0051] Step S2, the cooperative path oscillation information between the multi-modal is obtained in each round of recommendation feedback update of the advertisement strategy engine, and the multi-modal cooperative path oscillation coefficient is obtained according to the cooperative path oscillation information;

[0052] Step S3, the advertisement strategy cooperative / feedback double oscillation situation analysis model is constructed according to the confidence resonance runaway coefficient and the multi-modal cooperative path oscillation coefficient, the advertisement strategy cooperative / feedback double oscillation situation analysis index is obtained, and the un-restorable cooperative imbalance and state drift risk of the current cooperative system is evaluated;

[0053] Step S4, the advertisement mode confidence degree gating mechanism and the advertisement strategy feedback damping mechanism are constructed according to the un-restorable cooperative imbalance and state drift risk of the cooperative system, and the system state is reconstructed;

[0054] In step S1, first, multi-modal advertisement data sources such as user behavior logs, content semantic features, environmental context, social interaction are collected, and the confidence resonance runaway information of each mode in the response prediction process of putting is obtained, and the confidence resonance runaway coefficient is obtained according to the confidence resonance runaway information;

[0055] The confidence resonance runaway coefficient in the application is used for quantifying the cooperative drift degree caused by the dynamic guidance of the low-confidence mode by the high-confidence mode in the running process of the multi-modal system, is a core index for measuring the credibility and stability of multi-modal cooperation. The coefficient comprehensively considers the confidence gradient change between modes, the current mode state deviation degree and the dynamic characteristics of the system cooperative weight distribution, and can accurately reflect whether the low-confidence mode is forced to cooperate into the wrong judgment path without sufficient semantic support, and then form a "drift consensus". When the confidence resonance runaway coefficient is large, it indicates that there is a significant imbalance in the guidance of the current system, that is, some high-confidence modes cause passive resonance drift of weak modes due to weight dominance or historical feedback accumulation. This abnormal cooperation is often characterized by semantic consensus, but in essence, it hides the semantic conflict and cognitive difference between modal judgments. Once the wrong modal "consensus" becomes the input basis for subsequent reasoning and feedback optimization, not only will it amplify the system misjudgment risk, but also will significantly interfere with the convergence path of the model parameters, and finally induce the oscillation optimization or even irreversible deviation of the model.

[0056] On the contrary, when the confidence resonance runaway coefficient is small, it indicates that the current modal coordination structure of the system is stable, the weak mode is not guided by non-rationality, the coordination decision is more dependent on the semantic consistency within the mode, and the coordination mechanism reflects the real complementarity and information fusion, rather than the drifting consistency. This low resonance state reflects the system's strong ability to suppress false information sources in dynamic coordination, effectively maintaining each mode within its confidence boundary, and improving the overall robustness and anti-interference ability of the system.

[0057] The dynamic risk assessment mechanism based on the confidence resonance runaway coefficient can be used as a system-level coordination stability index to predict and intervene in the non-recoverable coordination imbalance and state drift risk in the coordination process;

[0058] Therefore, by obtaining the confidence resonance runaway information between multiple modes, the coordination drift caused by the dynamic guidance of the low-confidence mode by the high-confidence mode is analyzed, and the confidence resonance runaway coefficient is obtained to measure the degree of coordination drift caused by the dynamic guidance of the low-confidence mode by the high-confidence mode;

[0059] The logic of obtaining the confidence resonance runaway coefficient is as follows:

[0060] Obtain the state of each mode At time t, the mode state includes the semantic embedding vector and the confidence output , mark the mode state as ;

[0061] It should be noted that the semantic embedding vector can be obtained by feature extraction and high-dimensional semantic mapping of the original modal data by the modal encoder, and the confidence output refers to the confidence or probability score corresponding to the prediction output result of the current coordination decision task by the modal. It can be extracted by the classifier or decision maker head structure. The above semantic embedding extraction and confidence output acquisition method are mature technologies widely used in existing multi-modal systems, and will not be described here;

[0062] Calculate the confidence disturbance response value of the mode in the time interval : ;

[0063] Define the mode disturbance graph , and all modes constitute the node set of the mode disturbance graph , where is the number of modes, and if modes and mode satisfy: , then a directed edge is established, indicating potential resonance conduction;

[0064] It should be noted that in the above formula is the cosine similarity of semantic embedding vectors of modalities , , is a preset cosine similarity threshold, is a preset confidence perturbation response difference threshold;

[0065] A perturbation flow matrix is constructed on the modality perturbation graph, which is used to represent whether there is a perturbation of "low confidence modality guided by high confidence modality", wherein the elements of the perturbation flow matrix are: wherein is a perturbation state value, is a preset confidence threshold;

[0066] For low confidence modality , , the maximum length of its perturbed chain is obtained : wherein is a set of perturbation paths, representing all perturbation paths from any modality to modality as the final endpoint, is an edge in the perturbation path, representing one-hop perturbation conduction from modality to modality in path ;

[0067] A perturbation propagation factor matrix is constructed according to the confidence perturbation response value, wherein the elements of the perturbation propagation factor matrix are: wherein is a perturbation propagation factor, is the confidence perturbation response value of modality , is the path hop number, representing the propagation of perturbation from modality to modality in the path hop number of the modality perturbation graph;

[0068] A set of resonance closed loop paths existing in the modality perturbation graph is identified, and the resonance strength is calculated: ;

[0069] It should be noted that the resonance closed loop path is only constructed when the perturbation chain loop path exists (i.e. ): ;

[0070] In an optional example, the resonance strength is calculated as follows:

[0071] Assuming that the current modal number is 4, at time t, the confidence disturbance response value output values of the corresponding modes are respectively , , , ;

[0072] There is disturbance between its modes, that is: , , ;

[0073] And there is a resonance closed loop path ( );

[0074] Calculate the disturbance propagation factor: , , ;

[0075] Then the resonance strength is: ;

[0076] Calculate the confidence resonance out-of-control coefficient : , wherein is the average path length of each low-confidence mode Disturbed chain of resonance closed loop path: , The number of modes of the resonance closed loop path;

[0077] It should be noted that the above formulas are all dimensionless values, and the commonly used dimensionless methods include Min-Max normalization, Z-Score standardization, etc., which will not be repeated here;

[0078] Step S2, obtain the multi-modal collaborative path shock information through the multi-modal collaborative path shock information in each round of recommendation feedback (such as CTR response, click behavior, dwell time, etc.) update of the advertising strategy engine, and obtain the multi-modal collaborative path shock coefficient according to the collaborative path shock information;

[0079] The multimodal coordination path oscillation coefficient in the application is an important measurement index for quantitatively evaluating the dynamic stability of the end-to-end feedback regulation, and the core purpose is to reflect whether the multimodal coordination reasoning mechanism causes the coordination parameters to oscillate continuously in the closed-loop feedback path due to factors such as a dramatic change in the loss function, a design defect of the feedback path, or an imperfect modality regulation mechanism, and ultimately makes the system unable to converge, and even falls into an unresolvable coordination imbalance state of "oscillation-amplification-drift". The coefficient not only can dynamically monitor the steady-state behavior of the system in the training or reasoning stage, but also can early warn the potential dynamic out-of-control risk of the system, and has important guiding value for the stable operation and accuracy of the multimodal intelligent system. A larger multimodal coordination path oscillation coefficient usually means that the following system risk characteristics are accumulating: first, the parameters in the system feedback loop after multiple adjustments present high-frequency positive and negative swings, and the amplitude shows an amplification trend, indicating that the current feedback mechanism cannot effectively balance and adjust the error information; second, the coupling structure between modes is in a state of continuous disturbance, and cannot form convergent parameter synchronization; third, in the optimization process, the loss function shows significant non-monotonicity and instability, and the misleading information in the feedback path accumulates round by round, ultimately causing the model to fall into a "blind tuning" state and unable to achieve a stable optimal solution. Conversely, a smaller multimodal coordination path oscillation coefficient indicates that the system has good self-stabilization ability in the dynamic feedback process. Specifically, the model parameters tend to be stable after a limited number of feedback adjustments and no longer undergo large-scale repeated adjustments; the coordination coupling between modes gradually solidifies into a set of stable information paths and weight structures; the loss function shows good monotonicity in the training process, and the feedback mechanism does not produce interference amplification. At this time, the system shows strong steady-state coordination ability and is not easily affected by sudden mode drift or individual mode information deviation, and can effectively maintain the correctness and robustness of the overall task. Therefore, a significant reduction in the oscillation coefficient often means that the feedback path of the multimodal system has gradually converged to a set of robust optimal solution sets, the feedback regulation strategy is effective, and the overall operation of the system tends to be rational and stable.

[0080] The maximum beneficial effect of evaluating the system coordination imbalance and state drift risk based on the multimodal coordination path oscillation coefficient is that it provides a stability index system that is dynamically perceptible, parameter traceable, and decision intervenable for the entire end-to-end multimodal system. It can reveal whether the "internal operation logic of the model is stable" at a deeper level.

[0081] Therefore, by obtaining the coordination path oscillation information between multiple modes, the risk of continuous oscillation and non-convergence of modality coordination parameters in the closed loop due to unstable loss function or excessive feedback regulation in the end-to-end feedback adjustment is analyzed, and the multimodal coordination path oscillation coefficient is obtained to measure the risk degree of continuous oscillation and non-convergence of modality coordination parameters in the closed loop;

[0082] The logic for obtaining the oscillation coefficient of the multimodal cooperative path is as follows:

[0083] For mode Obtain the trajectory of collaborative parameters in consecutive feedback rounds , indicating the mode after the nth feedback. The collaborative fusion parameter vector (which can be fusion weights, attention vectors, gating factors, etc.);

[0084] Calculate the difference vector of continuous disturbance response : , , The total number of feedback rounds recorded;

[0085] Calculate the feedback response to the mode for each feedback based on the continuous disturbance response difference vector. rate of change of disturbance intensity of collaborative fusion parameters : ,in Let L2 norm be the difference vector of continuous disturbance responses. This is to prevent division by zero by a very small constant (generally taken as...). );

[0086] For modes Continuous disturbance response sequence Perform Fourier transform: ,in This is the frequency domain representation of a continuous perturbation response sequence. Calculate the frequency domain perturbation energy spectrum using Fourier transform. : ,in This is a high-frequency sub-interval (e.g., more than one-third of the frequency band above the sampling frequency) used to identify the intensity of high-frequency oscillations;

[0087] The non-convergent trend and "periodic oscillation instability" of the modes in the feedback are analyzed to obtain the local oscillation index of each mode. : :

[0088] For any mode pair ( , Record the oscillation synchronization index between modes : ,in For modality The continuous disturbance response difference vector;

[0089] The oscillation coefficient of the multimodal cooperative path is calculated based on the local oscillation index and the oscillation synchronization index. : ,in modal number;

[0090] It should be noted that the above formula is to calculate the dimensionless value, and the commonly used dimensionless method includes Min-Max normalization, Z-Score standardization, etc., which will not be repeated here;

[0091] Step S3, constructing an advertisement strategy synergy / feedback double shock situation analysis model according to the confidence resonance out-of-control coefficient and the multi-modal synergy path shock coefficient, obtaining an advertisement strategy synergy / feedback double shock situation analysis index, and evaluating the unresolvable synergy imbalance and state drift risk of the current synergy system;

[0092] According to the confidence resonance out-of-control coefficient and the multi-modal synergy path shock coefficient, an advertisement strategy synergy / feedback double shock situation analysis model is constructed, and an advertisement strategy synergy / feedback double shock situation analysis index is obtained. The formula on which the advertisement strategy synergy / feedback double shock situation analysis model is based is as follows , wherein is the advertisement strategy synergy / feedback double shock situation analysis index, is the confidence resonance out-of-control coefficient, is the multi-modal synergy path shock coefficient, respectively represent preset proportion coefficients of the confidence resonance out-of-control coefficient and the multi-modal synergy path shock coefficient, and are all greater than 0;

[0093] It should be noted that the above formula is to calculate the dimensionless value, and the commonly used dimensionless method includes Min-Max normalization, Z-Score standardization, etc., which will not be repeated here; According to the actual situation, for example, the expert weighting method is adopted, that is, experts in the relevant field are invited to determine the preset proportion coefficients of each index through professional opinion investigation and comprehensive evaluation, for example, may be 0.5, 0.5;

[0094] As can be seen from the above calculation expression, the greater the confidence resonance out-of-control coefficient and the multi-modal synergy path shock coefficient, the greater the advertisement strategy synergy / feedback double shock situation analysis index, indicating that the synergy state of the current multi-modal system is more unstable, and is at a higher risk of synergy imbalance and state drift edge, which may trigger a synergy misleading mode of non-convergence, error reinforcement or even "hallucinatory consensus" in the training or reasoning process of the system. On the contrary, the smaller the confidence resonance out-of-control coefficient and the multi-modal synergy path shock coefficient, the smaller the advertisement strategy synergy / feedback double shock situation analysis index, indicating that the system is currently in a low-risk and stable synergy situation, meaning that the synergy system has better convergence, coordination and robustness in the dynamic running process of end-to-end;

[0095] The advertisement strategy synergy / feedback double shock trend analysis index is compared with a preset advertisement strategy synergy / feedback double shock trend analysis index threshold value to evaluate the current synergy system's non-recoverable synergy imbalance and state drift risk, as follows:

[0096] If the advertisement strategy synergy / feedback double shock trend analysis index is greater than the advertisement strategy synergy / feedback double shock trend analysis index threshold value, it indicates that there is a high confidence resonance out-of-control risk and a synergy path feedback shock risk in the current system, and the two risk factors have jointly acted and amplified each other, which may lead the system to enter a non-recoverable synergy drift and non-convergence state, generating a high synergy imbalance and state drift risk.

[0097] If the advertisement strategy synergy / feedback double shock trend analysis index is less than or equal to the advertisement strategy synergy / feedback double shock trend analysis index threshold value, it indicates that the system as a whole is in a synergy stable interval, the current inter-modal cooperation is normal, the feedback adjustment mechanism is stable, the model convergence is good, and the synergy system still has self-regulation and abnormal absorption capacity, generating a low synergy imbalance and state drift risk.

[0098] It should be noted that the advertisement strategy synergy / feedback double shock trend analysis index threshold value can be preset according to specific application requirements, for example, a static setting method:

[0099] It can be set based on the maximum stable state index in the offline simulation training process.

[0100] Dynamic adaptive method:

[0101] In the online running of the system, the past N round trend index mean and fluctuation amplitude are calculated through a sliding window to update the threshold value in real time; it is suitable for scenes with drastic environmental changes or variable input distribution (such as cross-modal video stream perception systems);

[0102] Expert empowerment + experience calibration method:

[0103] Invite multi-modal modeling and deployment experts to set initial values according to task importance and system tolerance, and iteratively correct them combined with trial operation;

[0104] Step S4, according to the synergy system's non-recoverable synergy imbalance and state drift risk, an advertisement modal confidence gating mechanism and an advertisement strategy feedback damping mechanism are constructed to reconstruct the system state.

[0105] When a high synergy imbalance and state drift risk is generated, an advertisement modal confidence gating mechanism and an advertisement strategy feedback damping mechanism are constructed according to the current confidence resonance out-of-control coefficient and multi-modal synergy path shock coefficient to reconstruct the system state.

[0106] The advertisement modal confidence gating mechanism is as follows: wherein is the confidence output of the modal , is the reconstructed confidence output;

[0107] The advertisement strategy feedback damping mechanism is as follows: wherein is the Nth round of collaborative fusion parameter vector of the modal , is the reconstructed (N+1)th round of collaborative fusion parameter vector;

[0108] It should be noted that the above formulas are all dimensionless numerical calculations, and common dimensionless methods include Min-Max normalization, Z-Score standardization, etc., which will not be repeated here;

[0109] The present application dynamically captures the phenomenon of "irrational collaborative drift" of low-confidence modal being dragged by high-confidence modal in system collaborative reasoning by constructing a confidence resonance runaway coefficient, accurately revealing the "error consistency" risk in the system. Compared with the traditional static modal fusion method, this method is based on the joint modeling of modal confidence gradient and dynamic guiding effect, has stronger real-time perception ability and modal deviation sensitivity, can identify potential collaborative imbalance signs in advance before the system appears global misjudgment, secondly, combined with the multi-modal collaborative path shock coefficient constructed in each round of feedback optimization process, the convergence state and parameter disturbance trend of the system in the end-to-end feedback closed loop are quantified, the feedback shock behavior caused by misjudgment feedback, unstable loss function or out-of-control optimization step is found in time, and the confidence resonance runaway coefficient forms an advertisement strategy collaborative / feedback double shock situation analysis model, through the real-time output of the advertisement strategy collaborative / feedback double shock situation analysis index, the running state of the complex dynamic system is accurately quantitatively evaluated and judged, and based on the collaborative / feedback double shock situation analysis index, a modal confidence gate mechanism and a parameter feedback damping mechanism are further constructed to dynamically reconstruct the system state from two key aspects of collaborative signal entrance and feedback path.

[0110] Embodiment 2: This embodiment is an introduction to the end-to-end data dynamic collaborative system based on multi-modal data chain, as shown in Figure 2 , which includes a resonance runaway module, a collaborative path shock module, a risk comprehensive analysis module, and a state reconstruction module.

[0111] The resonance runaway module is used to obtain the confidence resonance runaway information of each modal in the prediction process of the response to the delivery, and obtain the confidence resonance runaway coefficient according to the confidence resonance runaway information.

[0112] A synergistic path oscillation module is configured to obtain synergistic path oscillation information between the multiple modes in each round of recommendation feedback update of the advertising strategy engine, and obtain a synergistic path oscillation coefficient of the multiple modes according to the synergistic path oscillation information.

[0113] A risk comprehensive analysis module is configured to construct an advertising strategy synergistic / feedback double oscillation situation analysis model according to the confidence resonance out-of-control coefficient and the synergistic path oscillation coefficient of the multiple modes, obtain an advertising strategy synergistic / feedback double oscillation situation analysis index, and evaluate the synergistic imbalance and state drift risk of the current synergistic system that is not recoverable.

[0114] A state reconstruction module is configured to reconstruct the system state according to the synergistic imbalance and state drift risk of the current synergistic system that is not recoverable.

[0115] The above formulas are all dimensionless values, and the formulas are obtained by collecting a large amount of data to simulate a formula of the most recent real situation. The preset parameters in the formula are set by a person skilled in the art according to the actual situation.

[0116] The above embodiments can be realized wholly or partially by software, hardware, firmware or any other combination. When realized by software, the above embodiments can be realized wholly or partially in the form of a computer program product. The computer program product includes one or more computer instructions or computer programs. When the computer instructions or computer programs are loaded or executed on a computer, the processes or functions described in the embodiments of the present application are wholly or partially generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network or other programmable devices. The computer instructions can be stored in a computer readable storage medium or transferred from one computer readable storage medium to another, for example, the computer instructions can be transferred from one website, computer, server or data center to another website, computer, server or data center through a wired or wireless (such as infrared, wireless, microwave, etc.) manner. The computer readable storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server, data center and the like containing one or more available medium collections. The available medium can be a magnetic medium (such as a floppy disk, a hard disk, a magnetic tape), an optical medium (such as a DVD) or a semiconductor medium. The semiconductor medium can be a solid state disk.

[0117] It should be understood that the size of the sequence number of each process described above in various embodiments of the present application does not mean the order of execution, and the execution order of each process should be determined according to its function and inherent logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0118] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the system and method described above can refer to the corresponding process in the foregoing method embodiment, and will not be described here.

[0119] In several embodiments provided in the present application, it should be understood that the disclosed system and method can be implemented in other ways.

[0120] The above merely provides a specific implementation of the present application, but the protection scope of the present application is not limited thereto, and any person skilled in the art can easily think of changes or replacements within the technical scope disclosed in the present application, which should be covered in the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. An end-to-end dynamic data collaboration method based on multimodal data links, characterized in that: Includes the following steps: Step S1: Obtain confidence resonance runaway information of each modality in the delivery response prediction process, and obtain the confidence resonance runaway coefficient based on the confidence resonance runaway information; Step S2: Obtain the multimodal collaborative path oscillation information in each round of recommendation feedback update of the advertising strategy engine, and obtain the multimodal collaborative path oscillation coefficient based on the collaborative path oscillation information; Step S3: Construct an advertising strategy collaboration / feedback dual oscillation situation analysis model based on the confidence resonance runaway coefficient and the multimodal collaborative path oscillation coefficient, obtain the advertising strategy collaboration / feedback dual oscillation situation analysis index, and assess the risk of irreversible collaboration imbalance and state drift in the current collaborative system. Step S4: Based on the risk of irreversible collaborative imbalance and state drift in the collaborative system, construct an advertising modality confidence gating mechanism and an advertising strategy feedback damping mechanism to reconstruct the system state; By acquiring confidence resonance runaway information of each modality in the delivery response prediction process, we analyze the collaborative drift of the low-confidence modality under the dynamic guidance of the high-confidence modality and obtain the confidence resonance runaway coefficient to measure the degree of collaborative drift of the low-confidence modality under the dynamic guidance of the high-confidence modality. The logic for obtaining the confidence resonance runaway coefficient is as follows: Get each modality At time t, the modal state includes a semantic embedding vector. and confidence output Mark the modal state as ; Calculate the mode in time interval Confidence perturbation response value ; Define modal perturbation diagram All modes constitute the node set of the modal perturbation graph. ,in Let be the number of modes, if the modes Modal Simultaneously satisfy: ,in The preset cosine similarity threshold, To establish a preset confidence level perturbation response difference threshold, a directed edge is created. ; A perturbation flow matrix is ​​constructed on the modal perturbation graph to indicate whether there is a perturbation "led by a high-confidence mode to a low-confidence mode". The elements of the perturbation flow matrix are: ,in The disturbance state value. This is a preset confidence threshold. For low confidence modes Obtain the maximum length of its disturbed chain. ; Construct a perturbation propagation factor matrix based on the confidence level perturbation response values, where the elements of the perturbation propagation factor matrix are: ,in As a disturbance propagation factor, For modality The confidence level perturbation response value, The path hop count represents the number of perturbations from mode. Propagation to modality Path hop count in the modal perturbation graph; Identify the set of resonant closed-loop paths present in the modal perturbation graph. Calculate the resonance intensity : ; Calculate the confidence level resonance runaway coefficient : ,in For each low-confidence mode of the resonant closed-loop path Average path length of the disturbed chain: , The number of modes in the resonant closed-loop path; By acquiring multimodal collaborative path oscillation information in each round of recommendation feedback update of the advertising strategy engine, we analyze the risk of modal collaborative parameters continuously oscillating and failing to converge in the closed loop due to unstable loss function or excessive feedback control during end-to-end feedback adjustment. We also obtain the multimodal collaborative path oscillation coefficient to measure the degree of risk of modal collaborative parameters continuously oscillating and failing to converge in the closed loop. The logic for obtaining the oscillation coefficient of the multimodal cooperative path is as follows: For mode Obtain the collaborative parameter trajectory in consecutive feedback rounds , indicating the mode after the nth feedback. The collaborative fusion parameter vector; Calculate the difference vector of continuous disturbance response : , , The total number of feedback rounds recorded; Calculate the feedback response to the mode for each feedback based on the continuous disturbance response difference vector. rate of change of disturbance intensity of collaborative fusion parameters ; For modes Continuous disturbance response sequence Perform Fourier transform: Calculate the frequency domain perturbation energy spectrum : ,in For high-frequency sub-intervals; The non-convergent trend and "periodic oscillation instability" of the modes in the feedback are analyzed to obtain the local oscillation index of each mode. : For any mode pair ( , Record the oscillation synchronization index between modes : ; The oscillation coefficient of the multimodal cooperative path is calculated based on the local oscillation index and the oscillation synchronization index. : ,in The modal number.

2. The end-to-end dynamic data collaboration method based on a multimodal data link according to claim 1, characterized in that: Based on the confidence resonance runaway coefficient and the multimodal collaborative path oscillation coefficient, an advertising strategy synergy / feedback dual oscillation trend analysis model is constructed to obtain the advertising strategy synergy / feedback dual oscillation trend analysis index. The formula used in the advertising strategy synergy / feedback dual oscillation trend analysis model is as follows: In the formula An index analyzing the dual oscillation trend of advertising strategy coordination and feedback. The confidence level resonance runaway coefficient is the coefficient. The multimodal cooperative path oscillation coefficient, represent the preset proportional coefficients of the confidence resonance runaway coefficient and the multimodal cooperative path oscillation coefficient, respectively, and All are greater than 0.

3. The end-to-end dynamic data collaboration method based on a multimodal data link according to claim 2, characterized in that: The advertising strategy synergy / feedback dual oscillation trend analysis index is compared with the preset advertising strategy synergy / feedback dual oscillation trend analysis index threshold to assess the risk of irreversible synergy imbalance and state drift in the current synergy system, as follows: If the advertising strategy synergy / feedback dual oscillation trend analysis index is greater than the advertising strategy synergy / feedback dual oscillation trend analysis index threshold, then a high risk of synergy imbalance and state drift will be generated. If the advertising strategy synergy / feedback dual oscillation trend analysis index is less than or equal to the advertising strategy synergy / feedback dual oscillation trend analysis index threshold, then a low synergy imbalance and state drift risk will be generated.

4. The end-to-end dynamic data collaboration method based on a multimodal data link according to claim 3, characterized in that: When high collaborative imbalance and state drift risks are generated, the system state is reconstructed by constructing an advertising modal confidence gating mechanism and an advertising strategy feedback damping mechanism based on the current confidence resonance runaway coefficient and multimodal collaborative path oscillation coefficient. The advertising modality confidence gating mechanism is as follows: ,in For modality The confidence output, The output is the confidence score after reconstruction; The feedback damping mechanism of the advertising strategy is as follows: ,in For modality The Nth round of collaborative fusion parameter vector, This is the reconstructed N+1 round collaborative fusion parameter vector.

5. An end-to-end dynamic data collaboration system based on a multimodal data link, used to implement the end-to-end dynamic data collaboration method based on a multimodal data link as described in any one of claims 1-4, characterized in that: It includes a resonance runaway module, a cooperative path oscillation module, a risk comprehensive analysis module, and a state reconstruction module; The resonance runaway module is used to obtain the confidence resonance runaway information of each modality in the delivery response prediction process, and to obtain the confidence resonance runaway coefficient based on the confidence resonance runaway information; The collaborative path oscillation module is used to obtain collaborative path oscillation information between multiple modalities in each round of recommendation feedback update of the advertising strategy engine, and to obtain the multimodal collaborative path oscillation coefficient based on the collaborative path oscillation information. The risk comprehensive analysis module is used to construct an advertising strategy collaboration / feedback dual oscillation situation analysis model based on the confidence resonance runaway coefficient and the multimodal collaborative path oscillation coefficient, obtain the advertising strategy collaboration / feedback dual oscillation situation analysis index, and assess the risk of irreversible collaboration imbalance and state drift of the current collaborative system. The state reconstruction module is used to reconstruct the system state by constructing an advertising modality confidence gating mechanism and an advertising strategy feedback damping mechanism based on the risk of irreversible collaborative imbalance and state drift in the collaborative system.

Images