Data processing method and device and electronic equipment

Abstract

The embodiment of the invention discloses a data processing method and device and electronic equipment, and the method comprises the steps: obtaining historical fusion fault data in a target application scene in response to a data fusion request initiated for the target application scene; the method comprises the following steps: analyzing historical fusion fault data in a target application scene, determining a resource utilization degree corresponding to the historical fusion fault data, and constructing a candidate system group set and a candidate resource group set in the target application scene based on the determined resource utilization degree; planning a fusion path for each candidate system group based on the fault probability predicted for each candidate system group to form a fusion path set; after to-be-fused multi-modal data is obtained, a target resource group can be selected from a candidate resource group set and a target fusion path can be selected from a fusion path set based on a fault prediction result of the multi-modal data, the to-be-fused multi-modal data is fused based on the target fusion path and the target resource group, and a fusion result is output.

Description

Technical Field

[0001] This invention relates to the field of computer technology, and in particular to a data processing method, apparatus, and electronic device. Background Technology

[0002] The visual network (VIN) is an intelligent sensing and interaction network built around visual information and utilizing technologies such as the Internet of Things (IoT) and 5G. The diverse scenarios and complex needs of the VIN mean that single-modal data is insufficient for sensing requirements in complex environments. For example, in autonomous driving scenarios, camera data alone is insufficient to handle interference from heavy rain or strong light. Therefore, this paper proposes integrating data from different modalities to overcome the limitations of single-modal approaches and enhance the system's perception, understanding, and decision-making capabilities in complex environments.

[0003] Currently, simple data splicing of multimodal data is insufficient for deep fusion, resulting in poor fusion performance as the fused multimodal data fails to adequately adapt to specific scenario requirements. Therefore, a technical solution is needed to effectively improve the fusion effect of multimodal data. Summary of the Invention

[0004] The purpose of this invention is to provide a technical solution that can improve the fusion effect of multimodal data.

[0005] To solve the above-mentioned technical problems, the embodiments of the present invention are implemented as follows: In a first aspect, an embodiment of the present invention provides a data processing method, the method comprising: Receive data fusion requests initiated for the target application scenario; In response to the data fusion request, historical fusion failure data under the target application scenario is obtained, and the resource utilization corresponding to the historical fusion failure data is determined; Based on the resource utilization corresponding to the historical fusion failure data, a set of candidate system groups and a set of candidate resource groups are constructed under the target application scenario. The failure probability corresponding to each candidate system group in the set of candidate system groups is determined, and a fusion path is planned for each candidate system group based on the failure probability to form a set of fusion paths. Acquire multimodal data to be fused in the target application scenario, perform fault prediction on the multimodal data, and obtain the fault prediction result of the multimodal data; Based on the fault prediction results, a target fusion path corresponding to the multimodal data is selected from the fusion path set, and a target resource group corresponding to the multimodal data is selected from the candidate resource group set. Based on the target fusion path and the target resource group, the multimodal data to be fused is fused, and the fusion result is output.

[0006] In a second aspect, embodiments of the present invention provide a data processing apparatus, the apparatus comprising: The analysis module is used to receive data fusion requests initiated for a target application scenario; in response to the data fusion request, it obtains historical fusion failure data under the target application scenario and determines the resource utilization corresponding to the historical fusion failure data; The processing module is used to construct a set of candidate system groups and a set of candidate resource groups under the target application scenario based on the resource utilization corresponding to the historical fusion failure data, determine the failure probability corresponding to each candidate system group in the set of candidate system groups, and plan a fusion path for each candidate system group based on the failure probability to form a fusion path set. The prediction module is used to acquire multimodal data to be fused in the target application scenario, perform fault prediction on the multimodal data, and obtain the fault prediction result of the multimodal data. The fusion module is used to select a target fusion path corresponding to the multimodal data from the fusion path set and a target resource group corresponding to the multimodal data from the candidate resource group set based on the fault prediction result, and to fuse the multimodal data to be fused based on the target fusion path and the target resource group, and output the fusion result.

[0007] Thirdly, embodiments of the present invention provide an electronic device, including a processor, a memory, and a computer program stored in the memory and executable on the processor, wherein the computer program, when executed by the processor, implements the steps of the data processing method provided in the above embodiments.

[0008] Fourthly, embodiments of the present invention provide a computer-readable storage medium storing a computer program, wherein the computer program, when executed by a processor, implements the steps of the data processing method provided in the above embodiments.

[0009] Fifthly, embodiments of the present invention provide a computer program product, including a computer program that, when executed by a processor, implements the steps of the data processing method provided in the above embodiments. Attached Figure Description

[0010] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0011] Figure 1This is a flowchart illustrating a data processing method according to an embodiment of the present invention; Figure 2 This is a schematic diagram illustrating the process of constructing a candidate system group set and a candidate resource group set under a target application scenario, as provided in another embodiment of the present invention. Figure 3 A flowchart illustrating the process of determining a fusion path set is provided in another embodiment of the present invention; Figure 4 A schematic diagram of a process for selecting a target resource group is provided in another embodiment of the present invention; Figure 5 A logical schematic diagram of a data processing method provided in another embodiment of the present invention; Figure 6 A flowchart illustrating a data processing method according to another embodiment of the present invention; Figure 7 This is a schematic diagram of the structure of a data processing device provided in another embodiment of the present invention; Figure 8 This is a schematic diagram of the structure of an electronic device provided in another embodiment of the present invention. Detailed Implementation

[0012] This invention provides a method, apparatus, and electronic device.

[0013] To enable those skilled in the art to better understand the technical solutions of this invention, the technical solutions of the embodiments of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this invention, and not all embodiments. Based on the embodiments of this invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of this invention.

[0014] As the scenarios and demands of the visual network become more diversified and complex, single-modal data can no longer meet the perception needs in complex environments. By integrating data from different modalities, the limitations of single-modal data can be overcome, improving perception, understanding, and decision-making capabilities in complex environments. However, current methods simply involve data stitching together of multimodal data, failing to achieve deep fusion. This results in the fused multimodal data not being well-suited to the specific needs of the scenario, leading to poor fusion performance.

[0015] To address this, embodiments of this specification provide a data processing method, apparatus, and device. By analyzing historical fusion failure data under a target application scenario, a set of candidate system groups and a set of candidate resource groups are constructed for the target application scenario. A fusion path is planned for each candidate system group in the candidate system group set, forming a fusion path set. Based on this, after obtaining the multimodal data to be fused, a target resource group can be selected from the candidate resource group set and a target fusion path can be selected from the fusion path set based on the fault prediction results of the multimodal data. Then, based on the target fusion path and the target resource group, the multimodal data to be fused is fused, and the fusion result is output. Accordingly, based on historical fusion failure data, a correlation logic between "data-resource-scenario" is constructed, enabling precise matching between multimodal data processing and scenario resource capabilities, improving the adaptability of fusion, and thus improving the fusion effect of multimodal data.

[0016] For specific processing details, please refer to the following embodiments.

[0017] like Figure 1 As shown, this embodiment of the invention provides a data processing method. The execution subject of this method can be a terminal device or a server. The terminal device can be a mobile terminal device such as a mobile phone, tablet computer, or smartwatch, or a terminal device such as a computer. The server can be an independent server or a server cluster composed of multiple servers. Specifically, the method may include the following steps: In step S102, a data fusion request initiated for the target application scenario is received.

[0018] In this context, a data fusion request refers to a request to integrate, analyze, and process data from different modalities. A target application scenario refers to a scenario where a cross-modal information fusion mechanism is built by integrating multiple types of data (such as visual, auditory, tactile, and sensor information) to achieve comprehensive and accurate perception and understanding of the environment, objects, or events. For example, target application scenarios could include urban traffic scenarios, autonomous driving scenarios, equipment inspection scenarios, and unmanned retail scenarios.

[0019] In implementation, a data processing system can be pre-deployed as the execution entity of this method. This data processing system can interact with application systems associated with the target application scenario and can process multimodal data from multiple target application scenarios. Data fusion requests can be initiated manually by technicians or automatically by the data processing system. For example, a request initiation command can be pre-set on the data processing system. When the request initiation command is triggered, the data fusion request is automatically initiated. The request initiation command can be triggered manually by technicians, triggered periodically according to a preset trigger cycle, or automatically triggered when a preset trigger condition is detected.

[0020] In step S104, in response to the data fusion request, historical fusion failure data under the target application scenario is obtained, and the resource utilization corresponding to the historical fusion failure data is determined.

[0021] Historical fusion failure data refers to data related to fusion failures in the historical fusion records of multimodal data. For example, historical fusion failure data may include, but is not limited to, failure occurrence time, failure type, and failure duration. Resource utilization corresponding to historical fusion failure data refers to the degree to which resources were effectively used in the historical fusion records; these resources include, but are not limited to, memory, CPU, and network bandwidth.

[0022] In practical applications, a target application scenario is associated with multiple application systems. For example, a smart security scenario may be associated with multiple application systems such as video analysis systems and behavior analysis systems. In response to a data fusion request, historical fusion failure data can be obtained from the multiple application systems associated with the target application scenario. The fusion of multimodal data relies on the resources of the application systems. Based on the obtained historical fusion failure data, the resource utilization rate of the application systems associated with this data can be statistically analyzed as the resource utilization rate corresponding to the historical fusion failure data.

[0023] In step S106, based on the resource utilization corresponding to historical fusion failure data, a set of candidate system groups and a set of candidate resource groups are constructed under the target application scenario. The failure probability corresponding to each candidate system group in the set of candidate system groups is determined, and a fusion path is planned for each candidate system group based on the failure probability to form a fusion path set.

[0024] In this scenario, multiple application systems within the target application scenario can provide the same resources, but the resources provided by these systems may not be entirely identical for a specific task. For example, the memory required to execute a fusion task within the target application scenario can be provided by application system A or application system B within the target application scenario; this is a flexible choice.

[0025] In implementation, the resource utilization corresponding to historical fusion failure data can be used to determine the usage of each resource in the fusion task of the target application scenario. Furthermore, based on the determined resource usage, available resources for each application system are allocated, resulting in multiple candidate resource groups. For example, in the target application scenario, if the associated application systems include A, B, and C, and the fusion task requires 60GB of memory, then candidate resource groups can be obtained as follows: a{Application system A provides 20GB of memory, application system B provides 40GB of memory}, b{Application system A provides 30GB of memory, application system B provides 30GB of memory}, c{Application system B provides 40GB of memory, application system C provides 20GB of memory}, etc. These multiple candidate resource groups can form a set of candidate resource groups for the target application scenario.

[0026] Based on this, candidate system groups can be inferred from the application systems contained in the candidate resource groups. For example, based on the above candidate resource groups a, b, and c, candidate system groups containing application system A and application system B, as well as candidate system groups containing application system B and application system C, can be inferred. Multiple candidate system groups can form a set of candidate system groups under the target application scenario.

[0027] After constructing the candidate system group set, the failure probability of each candidate system group in the set can be predicted based on historical fusion failure data. A failure probability set corresponding to each candidate system group is then constructed based on the predicted failure probabilities, containing the failure type and its corresponding failure probability. Furthermore, based on the failure probability set corresponding to each candidate system group, a fusion path with a lower failure probability is planned for each candidate system group, resulting in a fusion path set corresponding to the candidate system group set.

[0028] In step S108, multimodal data to be fused under the target application scenario is acquired, and fault prediction is performed on the multimodal data to obtain the fault prediction result of the multimodal data.

[0029] Multimodal data refers to data with different characteristics and representations collected from different sensors or methods within the same scene. Examples include text, images, audio, video, and sensor data. Fault prediction using multimodal data aims to determine the probability of various faults occurring within this data. The fault prediction results for multimodal data include a mapping relationship between fault types and fault probabilities.

[0030] In implementation, a fault prediction model can be pre-trained. After obtaining the multimodal data to be fused from the application system associated with the target application scenario, the fault prediction model can be used to predict the faults of the multimodal data, obtain the various fault types that may occur in the multimodal data and their corresponding fault probabilities, and form the fault prediction results of the multimodal data based on the predicted fault types and their corresponding fault probabilities.

[0031] In step S110, based on the fault prediction results, a target fusion path corresponding to the multimodal data is selected from the fusion path set, and a target resource group corresponding to the multimodal data is selected from the candidate resource group set. Based on the target fusion path and the target resource group, the multimodal data to be fused is fused, and the fusion result is output.

[0032] In implementation, fault prediction results can be used to understand the various fault types and corresponding probabilities that may occur in multimodal data. Based on these results, avoiding high-risk fault types can be a goal. A fusion path with a lower probability of fault and better suited to the current risk scenario can be selected from the fusion path set as the target fusion path for the multimodal data. Furthermore, avoiding high-risk fault types can continue to be a goal, and candidate resource groups with lower probability of fault can be selected from the candidate resource group set as the target resource group for the multimodal data. For example, if fault prediction results show high fusion fault rates during morning and evening rush hours on a certain road segment, a fusion path of "rapid preprocessing at edge nodes + deep fusion in the cloud" can be selected during peak hours.

[0033] Then, using the resources in the target resource group, the multimodal data to be fused is fused according to the target fusion path, and the fusion result is output.

[0034] This invention provides a data processing method that, in response to a data fusion request initiated for a target application scenario, acquires historical fusion failure data under the target application scenario and determines the resource utilization of the application systems associated with the historical fusion failure data. Based on the resource utilization, a set of candidate system groups and a set of candidate resource groups under the target application scenario are constructed. The failure probability corresponding to each candidate system group in the candidate system group set is determined, and a fusion path is planned for each candidate system group based on the failure probability to form a fusion path set. Multimodal data to be fused under the target application scenario is acquired, and failure prediction is performed on the multimodal data to obtain the failure prediction result. Based on the failure prediction result, a target resource group corresponding to the multimodal data is selected from the candidate resource group set, and a target fusion path corresponding to the multimodal data is selected from the fusion path set. Based on the target fusion path and the target resource group, the multimodal data to be fused is fused, and the fusion result is output. Accordingly, based on historical fusion failure data, a correlation logic between "data-resource-scenario" is constructed, which can achieve accurate matching between multimodal data processing and scenario resource capabilities, improve the adaptability of fusion, and thus improve the fusion effect of multimodal data.

[0035] In the above or following embodiments, the processing method for constructing the candidate system group set and candidate resource group set under the target application scenario based on resource utilization in step S106 can be varied. One optional processing method is provided below, such as... Figure 2 As shown, the specific steps may include steps S202 to S208.

[0036] In step S202, based on historical fusion fault data and resource utilization, the impact correlation degree under the target application scenario is determined. The impact correlation degree is used to characterize the relationship between fault type and resource utilization.

[0037] In implementation, the types of faults that may occur in the target application scenario and their corresponding probabilities can be inferred based on historical fusion fault data. The fault types and probabilities can be analyzed in depth with resource utilization to summarize the influence patterns of resource utilization, fault types and probabilities, which can be used as the influence correlation in the target application scenario.

[0038] For example, the impact correlation can be represented in the form of a mapping relationship. The impact correlation can include: when the CPU resource utilization exceeds 80%, the probability of a fault such as "insufficient computing power causing fusion lag" will increase by 30%. The impact correlation can reflect the relationship between resource utilization and fault type and fault probability.

[0039] In step S204, based on the resource utilization corresponding to the historical fusion fault data, multiple candidate systems are selected from the application systems associated with the target application scenario.

[0040] Among them, candidate systems refer to application systems in the target application scenario where the currently available resources can meet the resource utilization requirements.

[0041] During implementation, the available resources of each application system in the target application scenario can be counted, and the available resources of each application system can be compared with the resource utilization corresponding to the historical fusion failure data. If the available resources of each resource type in the application system are higher than the resource utilization corresponding to the historical fusion failure data, then the application system is identified as a candidate system.

[0042] In step S206, resources in multiple candidate systems are allocated based on the influence correlation degree and a preset resource threshold to obtain multiple resource groups, thereby constructing a set of candidate resource groups for the target application scenario.

[0043] In implementation, a resource threshold can be preset. The resource threshold represents the upper limit of the total effective resources. For example, the resource threshold can be set to 100 units. The "unit" of the resource threshold is a unified metric set for multiple resources. From the available resources, combinations of available resources from different application systems that are less than the resource threshold are selected. These combinations are regarded as candidate resource groups, and multiple candidate resource groups together form a candidate resource group set.

[0044] In step S208, multiple candidate system groups are obtained based on multiple candidate resource groups and the association between candidate resources and candidate systems, so as to construct a set of candidate system groups under the target application scenario.

[0045] In implementation, candidate system groups can be extracted from multiple allocated candidate resource groups. These candidate resource groups may correspond to the same or different candidate system groups. Since resources are allocated to specific application systems, once candidate resource groups that meet resource thresholds are identified, the providers of these resources can be grouped into candidate system groups based on the relationship between the candidate resources and candidate systems. For example, if a candidate resource group includes 60% computing power provided by application system A and 50% computing power provided by application system B, then the corresponding "application system A + application system B" is extracted to form a candidate system group.

[0046] In practical applications, based on the current availability of resources in the target application scenario, resources are allocated by influencing correlation, and then filtered by resource threshold constraints to finally obtain candidate system combinations. The core is to find a feasible solution for "resource adaptation and system collaboration" for multimodal data fusion, so as to ensure that the fusion task can be carried out efficiently and stably in the current video network scenario.

[0047] Based on this, there are various ways to determine the impact correlation of the target application scenario in step S202 based on historical fused fault data. The following is an optional processing method, which may include the following steps.

[0048] Step S2022: Determine the historical effect value of data fusion based on the resource utilization corresponding to the historical fusion results and historical fusion failure data; Step S2024: Obtain the ideal effect value of data fusion, and calculate the effect difference between the ideal effect value and the historical effect value; Step S2026: Based on the effect difference, historical fusion fault data, and resource utilization, determine the impact correlation degree used to characterize the relationship between fault type and resource utilization.

[0049] Among them, historical fusion results refer to the fusion results obtained by successfully fusing multimodal data in the target application scenario, while ideal effect value is a quantitative representation of the ideal fusion result of multimodal data in the target application scenario. Ideal effect value can be pre-set based on business needs and technical standards.

[0050] In implementation, based on the resource utilization corresponding to historical fusion failure data, the historical fusion results are evaluated, and the evaluation results are quantified to obtain the historical effect value of data fusion. Using a preset ideal effect value, the effect difference between the ideal effect value and the historical effect value is calculated. This effect difference measures the fusion effect of the historical fusion results; the smaller the effect difference, the better the fusion effect of the historical results. Historical failure data and effect differences can be integrated to construct an impact factor. The impact factor can be in textual form or quantified numerically. For example, historical fusion failure data might show that insufficient computing power caused fusion lag, and the effect difference might be an 8% difference in recognition accuracy. Linking these elements together forms a comprehensive description of the negative impact on the fusion process. The impact factor clearly indicates how a certain type of failure will cause deviation in the effect.

[0051] Based on this, we can delve deeper into the intrinsic relationship between influencing factors and resource utilization, extracting the correlation degree. This correlation degree can be determined through manual analysis or algorithmic analysis. For example, by analyzing a large number of performance differences, we found that when CPU resource utilization exceeds 80%, the probability of failures such as "insufficient computing power leading to fusion stuttering" increases by 30%. The correlation degree can reflect the relationship between resource utilization and failure type and failure probability.

[0052] In practical applications, by clarifying the influence of correlation on resource utilization status, fault type, and fault probability, we can provide correlation basis based on historical data for subsequent operations such as resource allocation and fault avoidance in the fusion process. This allows for more targeted adjustments to the multimodal data fusion process, effectively preventing faults caused by excessive resource utilization from affecting the fusion effect.

[0053] In the above or following embodiments, there are various ways to plan a fusion path for each candidate system group based on the failure probability to form a fusion path set in step S106. One optional processing method is provided below, such as... Figure 3 As shown, the specific steps may include S302 to S310.

[0054] In step S302, based on the influence relationship between different fault types in the fault probability set corresponding to the candidate system group, other fault types associated with each fault type are determined.

[0055] For each candidate system group, the probability of various possible failure types is predicted, and the interactions between these failure probabilities are analyzed to determine the influence relationships between different failure types. For example, an increase in the "computing power conflict failure probability" may increase the "data transmission incompatibility failure probability," while the "insufficient storage resource failure probability" may be unaffected by other failure probabilities. Therefore, "computing power conflict failure" and "data transmission incompatibility failure" can be correlated, that is, failure types affected by other failure probabilities can be linked.

[0056] In step S304, the fault type associated with other fault types is determined as the first fault type, and the fault type not associated with other fault types is determined as the second fault type, thus obtaining the first fault type set and the second fault type set corresponding to the candidate system group.

[0057] In step S302 above, the predicted fault types for each candidate system group were associated. Based on this, the multiple fault types predicted for the candidate system group can be divided into two types: "affected by other fault probabilities" and "not affected by other fault probabilities". The fault types affected by other fault probabilities are determined as the first fault type associated with other fault types, and the fault types not affected by other fault probabilities are determined as the second fault type not associated with other fault types.

[0058] For each candidate system group, a first fault type set is formed by combining at least one first fault type, and a second fault type set is formed by combining at least one second fault type. The fault types contained in the first fault type sets corresponding to different candidate system groups are not completely the same, and the fault probabilities contained are also not completely the same; similarly, the fault types contained in the second fault type sets corresponding to different candidate system groups are not completely the same, and the fault probabilities contained are also not completely the same.

[0059] In step S306, based on the first fault type set and the second fault type set corresponding to each candidate system group, the candidate system groups are sorted according to the fault probability corresponding to each fault type to obtain the first fault system group set or the second fault system group set corresponding to each fault type.

[0060] In implementation, after obtaining the first fault type set and the second fault type set corresponding to each candidate system group, a fault type can be randomly selected from the first fault type set or the second fault type set. The candidate system groups are then sorted according to the fault probability corresponding to this fault type in the first fault type set or the second fault type set corresponding to different candidate system groups, so as to obtain the first fault system group set or the second fault system group set corresponding to this fault type.

[0061] For example, the probability of network delay fault type in "System C + System D" increases by 20%, and the probability of network delay fault type in "System A + System B" increases by 15%. "System C + System D" and "System A + System B" can be sorted in descending order according to the fault probability to obtain the first set of fault system groups corresponding to the network delay fault type. This first set of fault system groups includes multiple candidate system groups arranged in order.

[0062] In step S308, the transmission distance corresponding to each candidate system group is determined, and a system group distance set containing the transmission distances corresponding to multiple candidate system groups is constructed.

[0063] Transmission distance refers to the distance traveled to complete network transmission between all candidate systems included in the candidate system group. Each candidate system group corresponds to a transmission distance, and the transmission distances corresponding to multiple candidate system groups included in the candidate system group set together form the system group distance set.

[0064] Furthermore, the transmission distances in the system group distance set can be sorted in descending order to obtain a system group distance set with order.

[0065] In implementation, the network transmission delay between candidate systems can also be used to represent the transmission distance. For example, if the candidate system group includes application system A and application system B, and application system A and application system B are deployed on different edge nodes, the physical distance between application system A and application system B can be converted into network transmission delay. The network transmission delay between application system A and application system B is 20ms, and the network transmission delay between application system C and application system D is 5ms. Arranging these values ​​in descending order yields the system group distance set with 20ms first and 5ms second.

[0066] In step S310, based on the system group distance set, the first fault system group set, and the second fault system group set, a fusion path is planned for each candidate system group to form a fusion path set.

[0067] In implementation, the system group distance set, the first fault system group set, and the second fault system group set can be integrated to form a path impact factor. Based on the path impact factor, a fusion path is planned for each candidate system group. The path impact factor can be understood as an "evaluation standard." Candidate system groups with rapid fault growth, high stability risk, and long system distance will reflect higher risk costs in the factor.

[0068] For example, for "Application System A + Application System B", considering factors such as fault growth, stability risk, and distance, the preferred path is "System A preprocessing data + transmission via low-latency link + deep integration with System B". For "Application System C + Application System D", due to their short distance and low stability risk, the path of "parallel processing + collaborative verification" can be adopted. These planned integration paths for each candidate system group are summarized to obtain an integration path information set, providing a risk-controlled and efficiency-adaptive execution plan for the subsequent actual integration process.

[0069] In practical applications, by predicting failure probabilities, analyzing risk interactions, ranking key factors, constructing influencing factors, and planning fusion paths, the failure risks of candidate system groups are transformed into quantifiable and rankable factors. These factors are then used to design more reasonable fusion paths for each combination, enabling multimodal fusion of the video network to consider both efficiency and proactively avoid failure risks, ensuring a more stable and reliable fusion process.

[0070] In the above or below embodiments, there are multiple ways to process the multimodal data to be fused based on the target fusion path and the target resource group in step S110 and output the fusion result. The following provides an optional processing method, which may specifically include the following steps S1102 to S1104.

[0071] In step S1102, the multimodal data is labeled with key data area, pre-validation data area and post-validation data area, and the three data area labeling result sets are output; In step S1104, the multimodal data to be fused is fused based on the target fusion path, the target resource group, and the three data area label result sets, and the fusion result is output.

[0072] Among them, the data contained in the multimodal data can be marked according to the importance of the data, and the multimodal data can be marked as "key data area" for the main support of fusion, "pre-assistance verification data" and "post-assistance verification data" for the auxiliary fusion, so as to clarify the role of each data in the fusion process based on the data marking.

[0073] For example, in the intelligent transportation video network scenario, assuming the combined data segment contains "intersection video frame + traffic flow sensor data", the vehicle recognition box in the video and the real-time traffic value in the sensor can be marked as the "key data area", the background image of the previous moment of the video frame and the historical baseline traffic flow of the sensor can be marked as the "pre-assistance verification data area", and the dynamic changes of the next moment of the video frame and the subsequent trend data of the sensor can be marked as the "post-assistance verification data area".

[0074] In implementation, the resource allocation in the target resource group can be utilized to first verify the accuracy of the current key data using the pre-assistance verification data according to the target fusion path, and then perform deep fusion by combining the post-assistance verification data, and finally output the fusion result.

[0075] In practical applications, by marking data areas, selecting fusion paths, allocating target resources, and executing fusion logic, multimodal data is layered into "key + auxiliary" categories. The optimal path is selected in combination with risk scenarios, enabling precise coordination of resources, data, and paths. This not only ensures the core value of the fusion results but also avoids fusion errors caused by fault risks through verification via auxiliary data areas, thereby effectively improving the quality of the output fusion results.

[0076] In the above or below embodiments, the processing method for selecting the target resource group corresponding to the multimodal data from the candidate resource group set based on the fault prediction result in step S110 can be varied. One optional processing method is provided below, such as... Figure 4 As shown, the specific steps may include steps S402 to S408.

[0077] In step S402, the multimodal data is segmented according to the fault prediction results to obtain a set of data segments.

[0078] In implementation, based on the fault prediction results, influencing factors related to the fault can be identified, such as time and semantics. Then, the multimodal data can be segmented according to these influencing factors, resulting in multiple data segments. These segments form a data segment set. Multimodal data segmentation can be done through equal division, random division, etc. For example, multimodal data can be equally divided according to time. By splitting a large amount of multimodal data into controllable small data segments, the probability of fault occurrence can be reduced.

[0079] In step S404, the data segments in the data segment set are recombined to obtain at least one combined data segment, and a combined data segment length set is constructed based on the data segment length corresponding to the at least one combined data segment.

[0080] In implementation, data segments in the data segment set can be combined. For example, data segments from the 1st second and the 2nd second can be combined, as can data segments from the 2nd second and the 3rd second. Each combination forms a combined data segment, thus covering the fusion requirements of different time dimensions. Based on at least one combined data segment, a combined data segment set can be constructed. Then, the data segment length of each combined data segment in the combined data segment set is calculated, forming a combined data segment length set. This set contains the mapping relationship between combined data segments and data segment lengths. For example, a combined data segment containing 2 seconds of video + 2 seconds of sensor data has a data segment length of 4.

[0081] In step S406, based on the fusion path set, the candidate resource groups in the candidate resource group set are sorted to obtain the candidate resource group sorting set, and the resource proportion critical threshold is set according to the candidate resource group sorting set.

[0082] The fusion path set contains the fusion paths planned for each candidate system group, and the candidate resource group set contains one or more candidate resource groups corresponding to each candidate system group.

[0083] In implementation, candidate system groups can be used as a link to match the fusion paths in the fusion path set with the candidate resource groups in the candidate resource group set. Based on the matching relationship, the candidate resource groups in the candidate resource group set are sorted according to the order of the fusion paths in the fusion path set to obtain the candidate resource group sorted set.

[0084] Then, by combining the candidate resource group ranking set, a critical threshold for resource proportion can be set. The critical threshold for resource proportion refers to the upper limit of resource allocation. For example, based on path requirements, it can be set that edge computing resources can carry up to 50% of the data segment combination length, and cloud resources can carry 30%, thus clarifying the upper limit of resource allocation.

[0085] In step S408, based on the resource proportion critical threshold and the data segment length contained in the combined data segment length set, a target resource group is selected from the candidate resource group set for each combined data segment, and the target resource group selected for at least one combined data segment is taken as the target resource group corresponding to the multimodal data.

[0086] In implementation, based on a critical threshold for resource proportion, combined data segments within a range of lengths can be matched with candidate resource groups in a candidate resource group set to ensure a suitable target resource group for each combined data segment. For example, shorter combined data segments with lower resource consumption can be matched with edge computing resources, while longer combined data segments with higher resource requirements can be matched with cloud collaborative resources. Ultimately, a suitable matching scheme is selected, which includes the matching relationship between the combined data segments and the target resource groups.

[0087] In practical applications, through resource sorting, risk prediction, data segmentation, combined statistics, threshold constraints, and correlation matching, abstract fusion resources and multimodal data are transformed into precisely matched combinations of "candidate resource groups - combined data segments." This not only aligns with path planning but also mitigates failure risks, achieving efficient collaboration between resources and data. It enables deep adaptation between data processing needs and resource supply capabilities, improving the rationality of resource allocation and thus effectively enhancing the fusion effect.

[0088] It should be noted that the number of resource proportion thresholds set in step S404 above can be one or more.

[0089] When there is only one critical threshold for resource proportion, the target resource group can be directly selected from the candidate resource group set for each combined data segment.

[0090] When the number of resource proportion critical thresholds is N (N>1), N candidate resource groups can be selected for each combined data segment from the candidate resource group set; then, based on the target fusion path selected from the fusion path set, the target resource group is selected from the N candidate resource groups selected for each combined data segment.

[0091] In this embodiment, before fusing the multimodal data to be fused, the selected target resource group can be verified. An optional verification method is provided below, such as... Figure 5 and Figure 6 As shown, this verification method may specifically include the following steps.

[0092] Step S602: Obtain the pre-built virtual target application environment; Step S604: Based on the target fusion path and target resource group, multimodal data is fused in the virtual target application environment to obtain the virtual fusion result. Step S606: Evaluate the virtual fusion result, obtain the virtual effect value corresponding to the virtual fusion result, and calculate the effect difference between the virtual effect value and the historical effect value; Step S608: Determine whether the effect difference is lower than a preset threshold. If the effect difference is not lower than the preset threshold, it indicates that the current fusion result does not meet the fusion requirements, and proceed to step S610. If the effect difference is lower than the preset threshold, it indicates that the current fusion result meets the fusion requirements, and proceed to step S612. Step S610: Reset the resource proportion critical threshold to redetermine the target resource group for multimodal data; Step S612: Based on the target fusion path and target resource group, fuse the multimodal data to be fused and output the fusion result.

[0093] This specification provides a data processing method that constructs a multimodal data fusion process covering "historical analysis, resource optimization, risk prediction, dynamic matching, and fusion execution." Starting with historical scenario data mining, it analyzes fusion failure operation data, resource utilization, and effect values ​​to construct influencing relationships, and optimizes the allocation and combination of available resources based on these relationships. It then predicts the probability of failure risks around candidate system groups, analyzes the growth and stability of risk data, and plans information fusion paths based on factors such as inter-system distance. Based on the pre-selected fusion path and candidate resource group set, it achieves precise matching of data and resources. It marks key, pre- and post-verification data areas for the matched data segments, and executes fusion processing based on failure risks and the determined fusion path. This process differs from traditional single resource allocation or simple fusion processes. Through multi-stage data association and dynamic optimization, it forms a unique technical path for multimodal fusion in visual network scenarios, ensuring fusion efficiency and quality.

[0094] The above describes the data processing method provided in the embodiments of this specification. Based on the same idea, the embodiments of this specification also provide a data processing device, such as... Figure 7 As shown.

[0095] The data processing device includes: an analysis module 701, a processing module 702, a prediction module 703, a matching module 704, and a fusion module 705, wherein: Analysis module 701 is used to receive a data fusion request initiated for a target application scenario; in response to the data fusion request, obtain historical fusion failure data under the target application scenario, and determine the resource utilization corresponding to the historical fusion failure data; The processing module 702 is used to construct a set of candidate system groups and a set of candidate resource groups under the target application scenario based on the resource utilization corresponding to the historical fusion failure data, determine the failure probability corresponding to each candidate system group in the set of candidate system groups, and plan a fusion path for each candidate system group based on the failure probability to form a fusion path set. The prediction module 703 is used to acquire the multimodal data to be fused in the target application scenario, perform fault prediction on the multimodal data, and obtain the fault prediction result of the multimodal data. The matching module 704 is used to select a target fusion path corresponding to the multimodal data from the fusion path set based on the fault prediction result, and to select a target resource group corresponding to the multimodal data from the candidate resource group set. The fusion module 705 is used to fuse the multimodal data to be fused based on the target fusion path and the target resource group, and output the fusion result.

[0096] In this embodiment of the specification, the processing module 702 is used for: Based on the historical fusion fault data and the resource utilization, the impact correlation degree under the target application scenario is determined. The impact correlation degree is used to characterize the correlation between fault type and resource utilization. Based on the resource utilization corresponding to the historical fusion fault data, multiple candidate systems are selected from the application systems associated with the target application scenario. Based on the aforementioned impact correlation and the preset resource threshold, resources in multiple candidate systems are allocated to obtain multiple resource combinations, thereby constructing a candidate resource group set for the target application scenario. Based on the multiple candidate resource groups and the association between candidate resources and candidate systems, multiple candidate system groups are obtained to construct a candidate system group set for the target application scenario.

[0097] In this embodiment of the specification, the processing module 702 is used for: Based on the resource utilization corresponding to the historical fusion results and the historical fusion failure data, the historical effect value of data fusion is determined; Obtain the ideal effect value of data fusion, and calculate the effect difference between the ideal effect value and the historical effect value; Based on the effect difference, the historical fusion fault data, and the resource utilization, the influence correlation degree used to characterize the relationship between fault type and resource utilization is determined.

[0098] In this embodiment of the specification, the processing module 702 is used for: Based on the influence relationship between different fault types in the fault probability set corresponding to the candidate system group, other fault types associated with each fault type are determined. The fault type associated with other fault types is determined as the first fault type, and the fault type not associated with other fault types is determined as the second fault type, thus obtaining the first fault type set and the second fault type set corresponding to the candidate system group; Based on the first fault type set and the second fault type set corresponding to each candidate system group, the candidate system groups are sorted according to the fault probability corresponding to each fault type to obtain the first fault system group set or the second fault system group set corresponding to each fault type. Determine the transmission distance corresponding to each candidate system group, and construct a system group distance set containing the transmission distances corresponding to multiple candidate system groups; Based on the system group distance set, the first fault system group set, and the second fault system group set, a fusion path is planned for each candidate system group to form a fusion path set.

[0099] In the embodiments described in this specification, the matching module 704 is used for: Based on the fault prediction results, the multimodal data is segmented to obtain a data segment set; The data segments in the data segment set are recombined to obtain at least one combined data segment, and a combined data segment length set is constructed based on the data segment length corresponding to the at least one combined data segment. Based on the fusion path set, the candidate resource groups in the candidate resource group set are sorted to obtain a candidate resource group sorting set, and a resource proportion critical threshold is set according to the candidate resource group sorting set. Based on the resource proportion critical threshold and the data segment lengths contained in the data segment length set, a target resource group is selected from the candidate resource group set for each combined data segment, and the target resource group selected for the at least one combined data segment is taken as the target resource group corresponding to the multimodal data.

[0100] In the embodiments described in this specification, the fusion module 705 is used for: The multimodal data is labeled with key data area, pre-validation data area and post-validation data area, and the three data area labeling result sets are output; Based on the target fusion path, the target resource group, and the three data area labeling result sets, the multimodal data to be fused is fused, and the fusion result is output.

[0101] This specification provides a data processing apparatus that analyzes historical fusion failure data in a target application scenario to construct a set of candidate system groups and a set of candidate resource groups for the target application scenario. A fusion path is planned for each candidate system group in the candidate system group set, forming a fusion path set. Based on this, after acquiring the multimodal data to be fused, a target resource group can be selected from the candidate resource group set and a target fusion path can be selected from the fusion path set based on the fault prediction results of the multimodal data. Then, based on the target fusion path and the target resource group, the multimodal data to be fused is fused, and the fusion result is output. Accordingly, based on historical fusion failure data, a correlation logic between "data-resource-scenario" is constructed, enabling precise matching between multimodal data processing and scenario resource capabilities, improving the adaptability of fusion, and thus improving the fusion effect of multimodal data.

[0102] The above describes the data processing apparatus provided in the embodiments of this specification. Based on the same concept, the embodiments of this specification also provide an electronic device, such as... Figure 8 As shown.

[0103] The electronic device can provide a terminal device or server, etc., for the above embodiments.

[0104] Electronic devices can vary considerably due to differences in configuration or performance. They may include one or more processors 801 and memories 802, with the memory 802 storing one or more application programs or data. The memory 802 may be temporary or persistent storage. The application programs stored in the memory 802 may include one or more modules (not shown), each module including a series of computer-executable instructions for the electronic device. Furthermore, the processor 801 may be configured to communicate with the memory 802 and execute the series of computer-executable instructions stored in the memory 802 on the electronic device. The electronic device may also include one or more power supplies 803, one or more wired or wireless network interfaces 804, one or more input / output interfaces 805, and one or more keyboards 806.

[0105] Specifically, in this embodiment, the electronic device includes a memory and one or more programs, wherein one or more programs are stored in the memory, and one or more programs may include one or more modules, and each module may include a series of computer-executable instructions for use in the electronic device, and is configured to be executed by one or more processors. The one or more programs include computer-executable instructions for performing the following: Receive data fusion requests initiated for the target application scenario; In response to the data fusion request, historical fusion failure data under the target application scenario is obtained, and the resource utilization corresponding to the historical fusion failure data is determined; Based on the resource utilization corresponding to the historical fusion failure data, a set of candidate system groups and a set of candidate resource groups are constructed under the target application scenario. The failure probability corresponding to each candidate system group in the set of candidate system groups is determined, and a fusion path is planned for each candidate system group based on the failure probability to form a set of fusion paths. Acquire multimodal data to be fused in the target application scenario, perform fault prediction on the multimodal data, and obtain the fault prediction result of the multimodal data; Based on the fault prediction results, a target fusion path corresponding to the multimodal data is selected from the fusion path set, and a target resource group corresponding to the multimodal data is selected from the candidate resource group set. Based on the target fusion path and the target resource group, the multimodal data to be fused is fused, and the fusion result is output.

[0106] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the electronic device embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions of the method embodiments.

[0107] This specification provides an electronic device that analyzes historical fusion failure data in a target application scenario to construct a set of candidate system groups and a set of candidate resource groups for the target application scenario. A fusion path is planned for each candidate system group in the candidate system group set, forming a fusion path set. Based on this, after acquiring the multimodal data to be fused, a target resource group can be selected from the candidate resource group set and a target fusion path can be selected from the fusion path set based on the fault prediction results of the multimodal data. Then, based on the target fusion path and the target resource group, the multimodal data to be fused is fused, and the fusion result is output. Accordingly, based on historical fusion failure data, a correlation logic between "data-resource-scenario" is constructed, enabling precise matching between multimodal data processing and scenario resource capabilities, improving the adaptability of fusion, and thus improving the fusion effect of multimodal data.

[0108] Furthermore, based on the above Figures 1 to 6 The method shown in this specification, along with one or more embodiments, also provides a storage medium for storing computer-executable instruction information. In one specific embodiment, the storage medium can be a USB flash drive, optical disc, hard disk, etc. When the computer-executable instruction information stored in the storage medium is executed by a processor, it can achieve the following process: Receive data fusion requests initiated for the target application scenario; In response to the data fusion request, historical fusion failure data under the target application scenario is obtained, and the resource utilization corresponding to the historical fusion failure data is determined; Based on the resource utilization corresponding to the historical fusion failure data, a set of candidate system groups and a set of candidate resource groups are constructed under the target application scenario. The failure probability corresponding to each candidate system group in the set of candidate system groups is determined, and a fusion path is planned for each candidate system group based on the failure probability to form a set of fusion paths. Acquire multimodal data to be fused in the target application scenario, perform fault prediction on the multimodal data, and obtain the fault prediction result of the multimodal data; Based on the fault prediction results, a target fusion path corresponding to the multimodal data is selected from the fusion path set, and a target resource group corresponding to the multimodal data is selected from the candidate resource group set. Based on the target fusion path and the target resource group, the multimodal data to be fused is fused, and the fusion result is output.

[0109] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the above-described storage medium embodiment is basically similar to the method embodiment, so the description is relatively simple; relevant parts can be referred to the description of the method embodiment.

[0110] This specification provides a storage medium that analyzes historical fusion failure data in a target application scenario to construct a candidate system group set and a candidate resource group set for the target application scenario. A fusion path is planned for each candidate system group in the candidate system group set, forming a fusion path set. Based on this, after obtaining the multimodal data to be fused, a target resource group can be selected from the candidate resource group set and a target fusion path can be selected from the fusion path set based on the fault prediction results of the multimodal data. Then, based on the target fusion path and the target resource group, the multimodal data to be fused is fused, and the fusion result is output. Accordingly, based on historical fusion failure data, a correlation logic between "data-resource-scenario" is constructed, enabling precise matching between multimodal data processing and scenario resource capabilities, improving the adaptability of fusion, and thus improving the fusion effect of multimodal data.

[0111] Furthermore, based on the above Figures 1 to 6The method shown in this specification, along with one or more embodiments, also provides a computer program product including a computer program that, when executed by a processor, performs the following process: Receive data fusion requests initiated for the target application scenario; In response to the data fusion request, historical fusion failure data under the target application scenario is obtained, and the resource utilization corresponding to the historical fusion failure data is determined; Based on the resource utilization corresponding to the historical fusion failure data, a set of candidate system groups and a set of candidate resource groups are constructed under the target application scenario. The failure probability corresponding to each candidate system group in the set of candidate system groups is determined, and a fusion path is planned for each candidate system group based on the failure probability to form a set of fusion paths. Acquire multimodal data to be fused in the target application scenario, perform fault prediction on the multimodal data, and obtain the fault prediction result of the multimodal data; Based on the fault prediction results, a target fusion path corresponding to the multimodal data is selected from the fusion path set, and a target resource group corresponding to the multimodal data is selected from the candidate resource group set. Based on the target fusion path and the target resource group, the multimodal data to be fused is fused, and the fusion result is output.

[0112] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the above-described embodiment of a computer program product is relatively simple in description because it is fundamentally similar to the method embodiment; relevant parts can be referred to the description of the method embodiment.

[0113] This specification provides a computer program product that analyzes historical fusion failure data in a target application scenario to construct a set of candidate system groups and a set of candidate resource groups for the target application scenario. It then plans a fusion path for each candidate system group in the candidate system group set, forming a fusion path set. Based on this, after obtaining the multimodal data to be fused, it can select a target resource group from the candidate resource group set and a target fusion path from the fusion path set based on the fault prediction results of the multimodal data. Then, based on the target fusion path and the target resource group, it fuses the multimodal data to be fused and outputs the fusion result. Accordingly, based on historical fusion failure data, it constructs a correlation logic between "data-resource-scenario," enabling precise matching between multimodal data processing and scenario resource capabilities, improving the adaptability of fusion, and thus improving the fusion effect of multimodal data.

[0114] The foregoing has described specific embodiments of this specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims may be performed in a different order than that shown in the embodiments and may still achieve the desired result. Furthermore, the processes depicted in the drawings do not necessarily require the specific or sequential order shown to achieve the desired result. In some embodiments, multitasking and parallel processing are possible or may be advantageous.

[0115] In the 1990s, improvements to a technology could be clearly distinguished as either hardware improvements (e.g., improvements to the circuit structure of diodes, transistors, switches, etc.) or software improvements (improvements to the methodology). However, with technological advancements, many methodological improvements today can be considered direct improvements to the hardware circuit structure. Designers almost always obtain the corresponding hardware circuit structure by programming the improved methodology into the hardware circuit. Therefore, it cannot be said that a methodological improvement cannot be implemented using hardware physical modules. For example, a Programmable Logic Device (PLD) (such as a Field Programmable Gate Array (FPGA)) is such an integrated circuit whose logic function is determined by the user programming the device. Designers can program and "integrate" a digital system onto a PLD themselves, without needing chip manufacturers to design and manufacture dedicated integrated circuit chips. Furthermore, nowadays, instead of manually manufacturing integrated circuit chips, this programming is mostly implemented using "logic compiler" software. Similar to the software compiler used in program development, the original code before compilation must also be written in a specific programming language, called a Hardware Description Language (HDL). There are many HDLs, such as ABEL (Advanced Boolean Expression Language), AHDL (Altera Hardware Description Language), Confluence, CUPL (Cornell University Programming Language), HDCal, JHDL (Java Hardware Description Language), Lava, Lola, MyHDL, PALASM, and RHDL (Ruby Hardware Description Language). Currently, the most commonly used are VHDL (Very-High-Speed ​​Integrated Circuit Hardware Description Language) and Verilog. Those skilled in the art should also understand that by simply performing some logic programming on the method flow using one of these hardware description languages ​​and programming it into an integrated circuit, the hardware circuit implementing the logical method flow can be easily obtained.

[0116] The controller can be implemented in any suitable manner. For example, it can take the form of a microprocessor or processor and a computer-readable medium storing computer-readable program code (e.g., software or firmware) executable by the (micro)processor, logic gates, switches, application-specific integrated circuits (ASICs), programmable logic controllers, and embedded microcontrollers. Examples of controllers include, but are not limited to, the following microcontrollers: ARC 625D, Atmel AT91SAM, Microchip PIC18F26K20, and Silicon Labs C8051F320. A memory controller can also be implemented as part of the control logic of the memory. Those skilled in the art will also recognize that, in addition to implementing the controller in purely computer-readable program code form, the same functionality can be achieved by logically programming the method steps to make the controller take the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers, and embedded microcontrollers. Therefore, such a controller can be considered a hardware component, and the means included therein for implementing various functions can also be considered as structures within the hardware component. Alternatively, the means for implementing various functions can be considered as both software modules implementing the method and structures within the hardware component.

[0117] The systems, devices, modules, or units described in the above embodiments can be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer. Specifically, a computer can be, for example, a personal computer, laptop computer, cellular phone, camera phone, smartphone, personal digital assistant, media player, navigation device, email device, game console, tablet computer, wearable device, or any combination of these devices.

[0118] For ease of description, the above apparatus is described by dividing it into various functional units. Of course, when implementing one or more embodiments of this specification, the functions of each unit can be implemented in one or more software and / or hardware.

[0119] Those skilled in the art will understand that the embodiments of this specification can be provided as methods, systems, or computer program products. Therefore, one or more embodiments of this specification may take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, one or more embodiments of this specification may take the form of a computer program product implemented on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0120] Embodiments in this specification are described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this specification. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable parallel device to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable parallel device, generate instructions for implementing the flowchart illustrations. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0121] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable fraud device to operate in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0122] These computer program instructions can also be loaded onto a computer or other programmable device, causing a series of operational steps to be performed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable device for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0123] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0124] Memory may include non-persistent storage in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0125] Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0126] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0127] Those skilled in the art will understand that the embodiments of this specification can be provided as methods, systems, or computer program products. Therefore, one or more embodiments of this specification may take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, one or more embodiments of this specification may take the form of a computer program product implemented on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0128] One or more embodiments of this specification can be described in the general context of computer-executable instructions, such as program modules, that are executed by a computer. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform a particular task or implement a particular abstract data type. One or more embodiments of this specification can also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communication network. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices.

[0129] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to interchangeably. Each embodiment focuses on describing the differences from other embodiments. In particular, the system embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions in the method embodiments.

[0130] The above description is merely an embodiment of this specification and is not intended to limit this document. Various modifications and variations can be made to this specification by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this specification should be included within the scope of the claims of this specification.

Claims

1. A data processing method, characterized in that, The method includes: Receive data fusion requests initiated for the target application scenario; In response to the data fusion request, historical fusion failure data under the target application scenario is obtained, and the resource utilization corresponding to the historical fusion failure data is determined; Based on the resource utilization corresponding to the historical fusion failure data, a set of candidate system groups and a set of candidate resource groups are constructed under the target application scenario. The failure probability corresponding to each candidate system group in the set of candidate system groups is determined, and a fusion path is planned for each candidate system group based on the failure probability to form a set of fusion paths. Acquire multimodal data to be fused in the target application scenario, perform fault prediction on the multimodal data, and obtain the fault prediction result of the multimodal data; Based on the fault prediction results, a target fusion path corresponding to the multimodal data is selected from the fusion path set, and a target resource group corresponding to the multimodal data is selected from the candidate resource group set. Based on the target fusion path and the target resource group, the multimodal data to be fused is fused, and the fusion result is output.

2. The method according to claim 1, characterized in that, The step of constructing a candidate system group set and a candidate resource group set under the target application scenario based on the resource utilization includes: Based on the historical fusion fault data and the resource utilization, the impact correlation degree under the target application scenario is determined. The impact correlation degree is used to characterize the correlation between fault type and resource utilization. Based on the resource utilization corresponding to the historical fusion fault data, multiple candidate systems are selected from the application systems associated with the target application scenario. Based on the aforementioned impact correlation and the preset resource threshold, resources in multiple candidate systems are allocated to obtain multiple resource combinations, thereby constructing a candidate resource group set for the target application scenario. Based on the multiple candidate resource groups and the association between candidate resources and candidate systems, multiple candidate system groups are obtained to construct a candidate system group set for the target application scenario.

3. The method according to claim 2, characterized in that, The determination of the impact correlation in the target application scenario based on the historical fusion fault data includes: Based on the resource utilization corresponding to the historical fusion results and the historical fusion failure data, the historical effect value of data fusion is determined; Obtain the ideal effect value of data fusion, and calculate the effect difference between the ideal effect value and the historical effect value; Based on the effect difference, the historical fusion fault data, and the resource utilization, the influence correlation degree used to characterize the relationship between fault type and resource utilization is determined.

4. The method according to claim 1, characterized in that, The step of planning a fusion path for each candidate system group based on the failure probability to form a fusion path set includes: Based on the influence relationship between different fault types in the fault probability set corresponding to the candidate system group, other fault types associated with each fault type are determined. The fault type associated with other fault types is determined as the first fault type, and the fault type not associated with other fault types is determined as the second fault type, thus obtaining the first fault type set and the second fault type set corresponding to the candidate system group; Based on the first fault type set and the second fault type set corresponding to each candidate system group, the candidate system groups are sorted according to the fault probability corresponding to each fault type to obtain the first fault system group set or the second fault system group set corresponding to each fault type. Determine the transmission distance corresponding to each candidate system group, and construct a system group distance set containing the transmission distances corresponding to multiple candidate system groups; Based on the system group distance set, the first fault system group set, and the second fault system group set, a fusion path is planned for each candidate system group to form a fusion path set.

5. The method according to claim 1, characterized in that, The step of selecting the target resource group corresponding to the multimodal data from the candidate resource group set based on the fault prediction result includes: Based on the fault prediction results, the multimodal data is segmented to obtain a data segment set; The data segments in the data segment set are recombined to obtain at least one combined data segment, and a combined data segment length set is constructed based on the data segment length corresponding to the at least one combined data segment. Based on the fusion path set, the candidate resource groups in the candidate resource group set are sorted to obtain a candidate resource group sorting set, and a resource proportion critical threshold is set according to the candidate resource group sorting set. Based on the resource proportion critical threshold and the data segment lengths contained in the data segment length set, a target resource group is selected from the candidate resource group set for each combined data segment, and the target resource group selected for the at least one combined data segment is taken as the target resource group corresponding to the multimodal data.

6. The method according to claim 1, characterized in that, The process of fusing the multimodal data to be fused based on the target fusion path and the target resource group, and outputting the fusion result, includes: The multimodal data is labeled with key data area, pre-validation data area and post-validation data area, and the three data area labeling result sets are output; Based on the target fusion path, the target resource group, and the three data area labeling result sets, the multimodal data to be fused is fused, and the fusion result is output.

7. A data processing apparatus, characterized in that, The device includes: The analysis module is used to receive data fusion requests initiated for a target application scenario; in response to the data fusion request, it obtains historical fusion failure data under the target application scenario and determines the resource utilization corresponding to the historical fusion failure data; The processing module is used to construct a set of candidate system groups and a set of candidate resource groups under the target application scenario based on the resource utilization corresponding to the historical fusion failure data, determine the failure probability corresponding to each candidate system group in the set of candidate system groups, and plan a fusion path for each candidate system group based on the failure probability to form a fusion path set. The prediction module is used to acquire multimodal data to be fused in the target application scenario, perform fault prediction on the multimodal data, and obtain the fault prediction result of the multimodal data. The matching module is used to select the target fusion path corresponding to the multimodal data from the fusion path set based on the fault prediction result, and to select the target resource group corresponding to the multimodal data from the candidate resource group set. The fusion module is used to fuse the multimodal data to be fused based on the target fusion path and the target resource group, and output the fusion result.

8. An electronic device, characterized in that, It includes a processor, a memory, and a computer program stored in the memory and executable on the processor, wherein the computer program, when executed by the processor, implements the steps of the data processing method as described in any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that, A computer program is stored on the computer-readable storage medium, which, when executed by a processor, implements the steps of the data processing method as described in any one of claims 1 to 6.

10. A computer program product, characterized in that, It includes a computer program that, when executed by a processor, implements the steps of the data processing method according to any one of claims 1 to 6.

Images