Cross-modal ai processing platform and methods thereof

Abstract

The application relates to the technical field of AI engine and discloses a cross-modal AI processing platform and a method thereof, wherein the method comprises the following steps: generating a relay context carrying a modal type set and a target model name according to a platform API key in a request header, and determining target channel configuration information and an alternative channel identifier from a registered channel in a channel capability table; performing message list field structured remapping, non-support parameter elimination and channel API key injection processing on a request body to obtain a target channel format request message and send the target channel format request message to a service endpoint in the target channel configuration information to obtain a downstream channel response message; and performing reverse structured remapping and streaming push data frame standardization processing to obtain a standard format response message and return the standard format response message to a request initiator, so that the technical problem of repeated development of multiple sets of protocol adaptation codes in the prior art is solved, and fine operation control of the cross-modal AI engine aggregation platform is realized.

Description

Technical Field

[0001] This invention relates to the field of AI engine technology, and in particular to a cross-modal AI processing platform and method. Background Technology

[0002] In the current artificial intelligence industry, multiple large model service providers maintain their own independent interface protocols, authentication systems, and response formats. When integrating multiple models, developers must develop protocol adaptation code separately for each service provider, resulting in high integration costs, long maintenance cycles, and any change to the interface of any service provider will trigger a chain of adaptation problems.

[0003] Existing request routing aggregation solutions are simple proxy forwarding schemes, capable only of recognizing text-based requests. They cannot identify and classify multimodal content such as images and audio within the request body. Consequently, they cannot select downstream channels with corresponding processing capabilities based on the request's modality, leading to frequent routing of multimodal requests to incompatible channels and resulting in call failures. Existing technologies lack a multidimensional dynamic routing mechanism that considers overall cost, performance, and quotas. They cannot automatically select the optimal channel based on the real-time health status and resource availability of each channel, nor do they possess failover capabilities to automatically switch to alternative channels when the primary channel fails. This results in low resource utilization, poor service availability, and difficulty in meeting the stable scheduling requirements of enterprise-level multimodal AI service aggregation. Summary of the Invention

[0004] This invention provides a cross-modal AI processing platform and method. This invention realizes bidirectional deep conversion between the platform and multiple heterogeneous protocol service providers. Developers only need to connect to a unified platform interface to transparently access all downstream channels, which solves the technical problem of repeated development of multiple protocol adaptation codes in the prior art.

[0005] In a first aspect, the present invention provides a cross-modal AI processing method, the cross-modal AI processing method comprising: Generate a relay context carrying a set of modality types and the target model name based on the platform API key in the request header, and determine the target channel configuration information and alternative channel identifiers from the registered channels in the channel capability table; Based on the target channel configuration information, the message list field structure remapping, non-supported parameter removal and channel API key injection processing are performed on the request body to obtain the target channel format request message, and the target channel format request message is sent to the service endpoint in the target channel configuration information to obtain the downstream channel response message; The downstream channel response message is subjected to reverse structured remapping and streaming push data frame standardization processing to obtain a standard format response message and return it to the request initiator.

[0006] In conjunction with the first aspect, in a first implementation of the first aspect of the present invention, the step of generating a relay context carrying a set of modality types and a target model name based on the platform API key in the request header, and determining the target channel configuration information and alternative channel identifiers from the registered channels in the channel capability table, includes: Perform a key-value matching query on the platform API key in the request header to obtain the token permission configuration; Based on the whitelist of allowed call models in the token permission configuration, the modal tag set of the request body and the target model name in the request body are written into the relay context. The modal tag set includes text modality, image modality and audio modality. Based on the relay context, the target channel configuration information and alternative channel identifiers are determined from the registered channels in the channel capability table.

[0007] In conjunction with the first aspect, in a second implementation of the first aspect of the present invention, determining the target channel configuration information and alternative channel identifiers from the registered channels in the channel capability table based on the relay context includes: Match the registration channels in the channel capability table that have a mapping record with the target model name in the relay context and are in the enabled state, and verify whether the registration channel capability tag field covers all modal types in the modal type set. The registration channels that pass the verification are determined as the candidate channel set. The target score of each candidate channel is calculated based on the unit price ratio of the pricing table model corresponding to each candidate channel in the candidate channel set, the recent average response latency in the request success rate statistics queue, and the remaining quota of the channel. The configuration information of the candidate channel with the highest target score is used as the target channel configuration information, and the candidate channel with the second highest target score is used as the alternative channel identifier.

[0008] In conjunction with the first aspect, in the third implementation of the first aspect of the present invention, the step of performing structured remapping of message list fields, removal of unsupported parameters, and channel API key injection processing on the request body based on the target channel configuration information to obtain a target channel format request message, and sending the target channel format request message to the service endpoint in the target channel configuration information to obtain a downstream channel response message, includes: Based on the channel type identifier in the target channel configuration information, the corresponding adapter plugin is loaded from the adapter registry, a structured remapping is performed on the message list fields of the request body, and the non-supported parameter fields in the request body are removed according to the adapter plugin to obtain the remapped request body; Based on the channel authentication mechanism type in the target channel configuration information, a channel API key is injected into the remapping request body to obtain a target channel format request message. The target channel format request message is then sent to the service endpoint in the target channel configuration information to obtain a downstream channel response message.

[0009] In conjunction with the first aspect, in the fourth implementation of the first aspect of the present invention, the step of loading the corresponding adapter plugin from the adapter registry based on the channel type identifier in the target channel configuration information, performing structured remapping on the message list fields of the request body, and removing unsupported parameter fields in the request body according to the adapter plugin to obtain a remapped request body includes: The corresponding adapter plugin is loaded from the adapter registry based on the channel type identifier in the target channel configuration information; Perform a role-by-role traversal and structured remapping on the message list fields of the request body, and perform a key-by-key comparison of all parameter field keys in the request body with the parameter support whitelist maintained by the adapter plugin. Remove parameter fields whose keys do not exist in the parameter support whitelist to obtain the remapped request body.

[0010] In conjunction with the first aspect, in the fifth implementation of the first aspect of the present invention, the step of injecting a channel API key into the remapping request body according to the channel authentication mechanism type in the target channel configuration information to obtain a target channel format request message, and sending the target channel format request message to the server endpoint in the target channel configuration information to obtain a downstream channel response message, includes: Obtain the channel authentication mechanism type from the target channel configuration information; When the channel authentication mechanism type is an authorization header type, the channel API key is injected into the authorization field of the request header to obtain the target channel format request message; when the channel authentication mechanism type is a query parameter type, the channel API key is concatenated to the query parameter of the server endpoint address to obtain the target channel format request message; when the channel authentication mechanism type is a custom header type, the authentication header is constructed according to the target channel's private authentication format to obtain the target channel format request message. The target channel format request message is sent to the server endpoint, and a downstream channel response message is returned.

[0011] In conjunction with the first aspect, in the sixth implementation of the first aspect of the present invention, if the status code of the returned downstream channel response message is a channel-side service exception type, then the alternative channel configuration information is read from the channel capability table using the alternative channel identifier, the channel API key injection process is re-executed on the remapping request body, and the message is resent to the alternative channel service endpoint to obtain the downstream channel response message.

[0012] In conjunction with the first aspect, in the seventh implementation of the first aspect of the present invention, the step of performing reverse structured remapping and streaming push data frame standardization processing on the downstream channel response message to obtain a standard format response message and returning it to the request initiator includes: Perform reverse structured remapping on the private fields carrying the model output content in the response body of the downstream channel response message, write the downstream channel push data stream of the streaming response into a standard format push data frame, and push the standard format push data frame to the request initiator. For each standard format push data frame in the downstream channel push data stream, a stream end signal detection is performed. When a private stream end marker of the downstream channel is detected, a stream termination marker is pushed to the request initiator to obtain a standard format response message, and the standard format response message is returned to the request initiator.

[0013] In conjunction with the first aspect, in the eighth implementation of the first aspect of the present invention, if the token consumption amount is missing in the usage field of the standard format response message, the token quantity is counted locally by the token segmenter on the input content of the request body and the output content of the response, and the statistical result is used to fill the token consumption amount. Based on the Token consumption, the input unit price ratio and output unit price ratio of the corresponding target model in the pricing table, and the user group ratio to which the user belongs in the relay context, the fee calculation is performed. An atomic balance deduction is performed on the account balance data table, and the call timestamp, user identifier, target channel identifier, model name, Token consumption, and fee details are written to the log table. The standard format response message is then returned to the request initiator.

[0014] Secondly, the present invention provides a cross-modal AI platform, the cross-modal AI platform comprising: The analysis module is used to generate a relay context carrying a set of modality types and the target model name based on the platform API key in the request header, and to determine the target channel configuration information and alternative channel identifiers from the registered channels in the channel capability table; The remapping module is used to perform structured remapping of message list fields, removal of unsupported parameters, and injection of channel API keys on the request body based on the target channel configuration information, so as to obtain a target channel format request message, and send the target channel format request message to the service endpoint in the target channel configuration information to obtain a downstream channel response message. The standardization processing module is used to perform reverse structured remapping and streaming push data frame standardization processing on the downstream channel response message to obtain a standard format response message and return it to the request initiator.

[0015] The technical solution provided by this invention achieves bidirectional deep conversion between the platform and multiple heterogeneous protocol service providers by dynamically loading adapter plugins based on channel type identifiers, extracting system roles and remapping image modal structures on the request body message list fields, and performing reverse field path mapping on the response body. Developers only need to connect to a unified platform interface to transparently access all downstream channels, eliminating the burden of redundant development and maintenance of multiple sets of protocol adaptation code. This invention also ensures that each request is routed to the channel with the best modal capability and comprehensive score by performing modal type traversal identification and writing it into the relay context on the request body content fields, combined with channel capability tag filtering and multi-dimensional weighted scoring and sorting. Furthermore, it automatically switches to the backup channel when the preferred channel is abnormal, effectively improving the routing accuracy and service continuity of multimodal requests. Finally, this invention ensures the billing integrity and data consistency of cross-channel calls by enabling local statistics from the word segmenter when the token consumption field is missing, performing refined cost calculation based on user group ratios, and atomically deducting the account balance, thus achieving refined operation and management of the cross-modal AI engine aggregation platform.

[0016] Other features and advantages of the invention will be set forth in the description which follows, and will be apparent in part from the description, or may be learned by practicing the invention. The objects and other advantages of the invention are realized and obtained in accordance with the structures particularly pointed out in the description, claims and drawings.

[0017] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, preferred embodiments are described below in detail with reference to the accompanying drawings. Attached Figure Description

[0018] Figure 1 This is a schematic diagram of one embodiment of the cross-modal AI processing method in this invention; Figure 2 This is a schematic diagram illustrating the determination of target channel configuration information and alternative channel identifiers in an embodiment of the present invention; Figure 3 This is a schematic diagram of structured remapping in an embodiment of the present invention; Figure 4 This is a schematic diagram of reverse structured remapping in an embodiment of the present invention; Figure 5 This is a schematic diagram of one embodiment of the cross-modal AI platform in this invention. Detailed Implementation

[0019] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0020] The terms "comprising" and "having," and any variations thereof, used in the embodiments of this invention are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or device that includes a series of steps or units is not limited to the steps or units listed, but may optionally include other steps or units not listed, or may optionally include other steps or units inherent to these processes, methods, products, or devices.

[0021] To facilitate understanding of this embodiment, a cross-modal AI processing method disclosed in this invention will first be described in detail. For example... Figure 1 As shown, this method includes the following steps: 101. Generate a relay context carrying a set of modal types and the target model name based on the platform API key in the request header, and determine the target channel configuration information and alternative channel identifiers from the registered channels in the channel capability table; 102. Based on the target channel configuration information, perform structured remapping of message list fields, removal of unsupported parameters, and injection of channel API keys on the request body to obtain the target channel format request message. Then, send the target channel format request message to the service endpoint in the target channel configuration information to obtain the downstream channel response message. 103. Perform reverse structured remapping and streaming push data frame standardization processing on the downstream channel response messages to obtain standard format response messages and return them to the request initiator.

[0022] In one specific embodiment, such as Figure 2 As shown, the process of executing step 101 can specifically include the following steps: 1011. Perform a key-value matching query on the platform API key in the request header to obtain the token permission configuration; 1012. Based on the whitelist of allowed call models in the token permission configuration, write the modal tag set of the request body and the target model name in the request body into the relay context. The modal tag set includes text modality, image modality and audio modality. 1013. Based on the relay context, determine the target channel configuration information and alternative channel identifiers from the registered channels in the channel capability table.

[0023] Specifically, upon receiving an external call request, the access layer extracts the platform API key from the request header and uses it as a query key in the token verification process to perform a key-value matching query. This query retrieves the corresponding token record from the token cache structure to quickly confirm whether the platform API key is valid, has a bound user, is still valid, or has been disabled. If the cache misses, the access layer then checks the persistent token data table to retrieve the user identifier, allowed call model whitelist, quota limit, and access control parameters corresponding to the platform API key, thus configuring the token permissions. If the platform API key is missing, expired, or disabled, the access layer terminates the current processing, preventing unauthorized requests from entering the model routing stage and reducing the pressure of invalid accesses on the backend channel side.

[0024] The parsing unit performs structured decomposition of the request body, reading the target model name and content units in the message list, and identifies the modal type contained in the request based on the organization of the message content. When the message content is plain text, it is marked as a text modal; when the message content contains image objects or image address fields, it is marked as an image modal; when the message content contains audio-related fields, it is marked as an audio modal. The identified modal tag set is jointly verified with the target model name in the request body, where the target model name must fall within the coverage of the allowed model whitelist. After successful verification, it is written into the relay context, so that the relay context simultaneously carries three key semantics: permission boundaries, model pointers, and modal features. Based on the relay context, the channel selection unit reads the target model name and modal type set from the relay context, and compares them with the registered channel records in the channel capability table to obtain the target channel configuration information and alternative channel identifiers.

[0025] In one specific embodiment, the process of performing step 1013 may specifically include the following steps: (1) Match the registered channels that have a mapping record with the target model name in the relay context and whose status is enabled from the channel capability table, and verify whether the registered channel capability flag field covers all modal types in the modal type set. The registered channels that pass the verification are determined as the candidate channel set. (2) Calculate the target score of each candidate channel based on the unit price ratio of the pricing table model corresponding to each candidate channel in the candidate channel set, the recent response delay average in the request success rate statistics queue, and the remaining quota of the channel. Use the configuration information of the candidate channel with the highest target score as the target channel configuration information and the candidate channel with the second highest target score as the alternative channel identifier.

[0026] Specifically, the routing and scheduling unit reads the target model name and modality type set from the relay context, then sends a capability query request to the channel management service. It then filters out all registered channels that are enabled and have established target model name mappings from the channel capability table. After the first round of filtering, it checks whether the capability tag field of each registered channel can completely cover all modality types carried in the current request. For example, if the request contains both text and image modalities, only registered channels with both text processing and visual understanding capabilities can be retained. If the request further includes audio modalities, it is necessary to further verify whether the audio processing capability tag exists. After this round of model mapping verification and modality coverage verification, the remaining channels can be determined as the candidate channel set.

[0027] The routing scheduling unit first performs availability threshold verification on the candidate channel set, and then performs comprehensive scoring and ranking. During the availability threshold verification phase, the unit reads the recent request success rate, timeout rate, and number of consecutive failures for each candidate channel. When the recent request success rate is lower than a preset availability threshold, or the number of consecutive failures reaches a preset cutoff threshold, the corresponding candidate channel is removed from the current scheduling set. For candidate channels that pass the availability threshold verification, the unit reads the model-converted unit price, recent average response latency, and remaining channel quota from the pricing table, and performs standardized processing on cost indicators, latency indicators, and quota indicators under a unified dimension. Then, a comprehensive score result is generated according to preset weights. After the comprehensive score is generated, the routing scheduling unit sorts the results in descending order and outputs the complete configuration information corresponding to the first-ranked candidate channel as the target channel configuration information. When a second candidate channel exists in the ranking result, the second-ranked candidate channel is registered as a backup channel identifier, which is used to trigger automatic switching when the first-ranked channel experiences service anomalies, network timeouts, or connection establishment failures before the first valid business data output.

[0028] In one specific embodiment, such as Figure 3 As shown, the process of executing step 102 can specifically include the following steps: 1021. Based on the channel type identifier in the target channel configuration information, load the corresponding adapter plugin from the adapter registry, perform structured remapping on the message list fields of the request body, and remove the non-supported parameter fields in the request body according to the adapter plugin to obtain the remapped request body. 1022. Based on the channel authentication mechanism type in the target channel configuration information, inject the channel API key into the remapping request body to obtain the target channel format request message, and send the target channel format request message to the service endpoint in the target channel configuration information to obtain the downstream channel response message.

[0029] Specifically, the process reads the channel type identifier from the target channel configuration information and uses it as the adapter retrieval key to locate the adapter plugin corresponding to the current channel protocol from the adapter registry. The adapter execution engine then loads the plugin, allowing request conversion to no longer rely on a fixed, hard-coded single path but instead invoke the corresponding conversion logic based on the protocol differences of different channels. After plugin loading, the processing chain performs structured remapping on the message list fields in the request body. Following the target channel's field organization rules, it reorganizes the message roles, historical rounds, system prompts, and multimodal content carrying structures. For example, system role messages can be extracted from the original message list and written into independent fields required by the target channel; historical dialogue content is rearranged according to the array structure specified by the target channel; and image-related content is converted according to the object format accepted by the target channel. The resulting request body retains the original request intent semantically, but its field format is now consistent with the target channel's interface specification. Meanwhile, the adapter plugin performs a key-by-key check on the parameter fields in the request body, comparing all parameter field key names with the parameter support whitelist maintained internally by the adapter. Parameters that are not supported by the target channel are directly removed, and parameters that are supported but whose values ​​exceed the limits of the target channel are trimmed and corrected, thereby generating a remapped request body that can be stably received by the target channel.

[0030] Different key injection methods are selected based on the channel authentication mechanism type in the target channel configuration information. When the channel uses an authorization header authentication mechanism, the channel API key is written into the authorization field of the request header; when the channel uses a query parameter authentication mechanism, the channel API key is appended to the query parameter part of the server endpoint address; when the channel uses a custom header authentication mechanism, the corresponding authentication header is generated according to the private authentication format agreed upon by the target channel. After the authentication information is injected, the remapping request body is further serialized into the content format required by the target channel, and encapsulated together with the authentication header and the server endpoint address into a complete target channel format request message. This message is then delivered to the server endpoint recorded in the target channel configuration information through the platform-side sending component, and the server endpoint returns the corresponding downstream channel response message.

[0031] In one specific embodiment, the process of performing step 1021 may specifically include the following steps: (1) Load the corresponding adapter plugin from the adapter registry based on the channel type identifier in the target channel configuration information; (2) Perform role-by-role traversal and structured remapping on the message list fields of the request body, and perform key-by-key comparison of all parameter field keys in the request body with the parameter support whitelist maintained by the adapter plugin. Remove parameter fields whose keys do not exist in the parameter support whitelist to obtain the remapped request body.

[0032] Specifically, the adaptation execution unit parses the channel type identifier in the target channel configuration information and uses it as a unique loading index to send to the adapter registry for plugin retrieval. The adapter registry pre-maintains the mapping relationship between different channel types and their corresponding adapter implementations. Therefore, the adaptation execution unit can directly locate the corresponding adapter plugin entry based on the channel type identifier, and then complete the instantiation, loading, and runtime binding. Thus, the adapter plugins corresponding to different channels respectively undertake the task of resolving protocol differences.

[0033] The adapter plugin performs a role-by-role traversal of the message list fields in the request body. During the traversal, it first identifies the role attributes corresponding to each message, and then performs structured remapping based on the target channel's private protocol structure. For channels that need to extract system prompts separately, the adapter plugin extracts the system role messages from the message list and rewrites them as top-level fields. For channels that need to distinguish between the latest question and historical rounds, the adapter plugin converts historical messages into a time-series array required by the target channel and rewrites the last round's user input into the specified carrier field. For cases where the message content contains image objects, the adapter plugin rewrites the image address, image source type, and multimodal content unit into a compatible data structure according to the target channel's requirements. After the role-by-role traversal, the session semantics, role order, and multimodal content in the original unified format request body remain consistent, but the field layout, node hierarchy, and content encapsulation method have been transformed into a private format that the target channel can directly parse, thereby eliminating the structural differences in message organization between different model service interfaces.

[0034] While the message list is being remapped, the adapter plugin performs a key-by-key comparison of all parameter fields in the request body, matching each parameter field key against the parameter support whitelist maintained internally by the adapter plugin. If a parameter field is not in the whitelist, it means the target channel does not accept that parameter, and the adapter plugin will directly remove it. If a parameter field is supported by the target channel, but its value exceeds the interface constraints of the target channel, the adapter plugin will further perform value pruning or normalization correction. The remapped request body only retains parameter fields that are truly supported by the target channel and have a valid format, effectively avoiding interface validation failures, call exceptions, or response errors caused by redundant parameters, illegal parameters, or residual cross-protocol inherited parameters.

[0035] In one specific embodiment, the process of performing step 1022 may specifically include the following steps: (1) Obtain the channel authentication mechanism type from the target channel configuration information; (2) When the channel authentication mechanism type is the authorization header type, the channel API key is injected into the authorization field of the request header to obtain the target channel format request message; when the channel authentication mechanism type is the query parameter type, the channel API key is concatenated to the query parameter of the server endpoint address to obtain the target channel format request message; when the channel authentication mechanism type is the custom header type, the authentication header is constructed according to the target channel private authentication format to obtain the target channel format request message. (3) Send the target channel format request message to the server endpoint and return the downstream channel response message.

[0036] Specifically, the adaptation execution unit reads the channel authentication mechanism type from the target channel configuration information and writes it into the sending context of this request as the selection criterion for the authentication injection branch. After the authentication mechanism is identified, if the channel authentication mechanism type is an authorization header type, the adaptation execution unit generates an authorization field in the request header and writes the channel API key into the authorization field according to the authorization format required by the target channel, enabling the channel to directly complete identity verification when receiving the message; if the channel authentication mechanism type is a query parameter type, the adaptation execution unit concatenates the channel API key to the query parameter segment of the server endpoint address, enabling the server endpoint address to have authentication capabilities while maintaining the original access path; if the channel authentication mechanism type is a custom header type, the adaptation execution unit constructs an authentication header according to the target channel's private authentication format and encapsulates the channel API key, authentication prefix, and necessary header key names into the request header set. The remapped request body is then bound to the authentication information, forming a target channel format request message that can be directly recognized by the target channel.

[0037] The sending control unit serializes the remapped request body according to the target channel requirements. It then encapsulates the serialized request body, the request header set with injected authentication information, and the server endpoint address from the target channel configuration information into a complete sending object. This object is then sent to the server endpoint via the platform-side sending component, initiating a downstream call. Upon receiving the target channel format request message, the server endpoint returns the corresponding downstream channel response message.

[0038] In one specific embodiment, if the status code of the returned downstream channel response message is a channel-side service exception type, the alternative channel configuration information is read from the channel capability table using the alternative channel identifier, the channel API key injection process is re-executed on the remapping request body, and the message is resent to the alternative channel service endpoint to obtain the downstream channel response message.

[0039] Specifically, after receiving the downstream channel response message from the preferred channel, the response receiving unit performs an exception type determination on the response status code. When the response status code is identified as a channel-side service exception, it is considered that although the preferred channel has completed request reception, it has failed to generate a valid response that can continue processing at the service availability level. In this case, the entire call is not terminated directly, but the primary / backup switchover link is triggered. After the primary / backup switchover link is triggered, the scheduling and control unit reads the backup channel configuration information from the channel capability table based on the backup channel identifier determined in the previous routing stage. The backup channel configuration information includes the channel type identifier, service endpoint address, and channel API key required for authentication injection corresponding to the backup channel. This allows the request, which has already been sent once, to switch to another downstream channel with execution capability for continued processing without changing the unified access semantics.

[0040] After the alternative channel configuration information is read, the adaptation execution unit directly calls the remapping request body and re-executes the channel API key injection process for the alternative channel. Since different channels may use different authentication mechanisms such as authorization headers, query parameters, or custom headers, the re-injection process constructs authentication fields according to the alternative channel's own authentication mechanism type, rebinding the remapping request body to the alternative channel's authentication rules to form a sending message consistent with the alternative channel's protocol. The sending control unit re-delivers the adapted request to the alternative channel's service endpoint, which returns a new downstream channel response message and sends it to the response standardization processing link. If the preceding anomaly is due to network timeout, channel overload, or connection establishment failure, and the primary channel has not yet output valid business content to the request initiator, the above primary / backup switching method can automatically switch channels while maintaining unified access semantics. This reduces the probability of the entire call chain being interrupted due to short-term instability of a single channel and improves stability and fault tolerance in multi-channel aggregation scenarios.

[0041] In one specific embodiment, such as Figure 4 As shown, the process of executing step 103 can specifically include the following steps: 1031. Perform reverse structured remapping on the private fields that carry the model output content in the response body of the downstream channel response message, write the downstream channel push data stream of the streaming response into a standard format push data frame, and push the standard format push data frame to the request initiator. 1032. Perform flow end signal detection on each standard format push data frame in the downstream channel push data stream. When the downstream channel private flow end marker is detected, push the stream termination marker to the request initiator, obtain the standard format response message, and return the standard format response message to the request initiator.

[0042] Specifically, the standardized processing unit performs reverse structured remapping on the private fields carrying the model output content in the response body, based on the response protocol rules corresponding to the current downstream channels. The key to reverse structured remapping is to restore the custom output fields, usage fields, and streaming incremental fields of different channels back to the platform's unified response structure, ensuring that the request initiator always faces the same standard format without needing to perceive the differences in field naming, hierarchical organization, and event encapsulation methods of the underlying channels. For non-streaming responses, the private fields carrying the output text are extracted and written into the message content position in the unified response structure, while the usage fields, model identifiers, and time information returned by the downstream channels are organized into the standard response object. For streaming responses, the data stream continuously pushed by the downstream channels is parsed frame by frame, extracting the newly added text fragments, incremental content, or event payloads in each frame, and then writing them into standard format push data frames, which are continuously pushed to the request initiator in the order of receipt. After the first standard format push data frame is output to the request initiator, subsequent processing links no longer perform channel rerouting, but only continue to complete the remaining frame transcription, stream end detection, and abnormal termination marker output.

[0043] During the continuous output of standard format push data frames, the standardization processing unit performs stream end signal detection on each standard format push data frame in the downstream channel push data stream. That is, while transcribing and forwarding frame by frame, it simultaneously determines whether the current frame contains a private stream end marker used by the downstream channel. Since different channels express stream end in different ways—some use dedicated event types to indicate end, and others use specific data segments—the detection logic needs to first identify the original end semantics according to channel adaptation rules, and then convert it into a unified stream termination marker on the platform side before pushing it to the request initiator. When a private stream end marker from a downstream channel is detected, the standardization processing unit stops receiving subsequent incremental content, sends a unified termination frame to the request initiator, and merges all the standard format push data frames that have been previously processed frame by frame, along with the termination marker, into a standard format response message, which is then returned to the request initiator by the access layer.

[0044] In one specific embodiment, if the token consumption is missing in the usage field of the standard format response message, the token quantity is counted locally by the token segmenter on the input content of the request body and the output content of the response, and the statistical result is used to fill the token consumption. Based on the token consumption, the input unit price ratio and output unit price ratio of the corresponding target model in the pricing table, and the user group ratio to which the user belongs in the relay context, the fee calculation is performed. An atomic balance deduction is performed on the account balance data table, and the call timestamp, user identifier, target channel identifier, model name, token consumption and fee details are written to the log table. A standard format response message is returned to the request initiator.

[0045] Specifically, the standardized processing unit checks whether the usage field in the standard format response message has completely returned the input and output token quantities. If the downstream channel has returned complete usage information, the usage information is directly used as the basis for billing and continues to flow. If the downstream channel has not returned complete usage information, or in a streaming scenario only incremental content is returned without carrying the final usage data, a local token statistics fallback link is activated. The input content of the request body and the output content of the response are respectively sent to the token segmenter to perform local token quantity statistics, and then the obtained statistical results are filled back into the usage field of the standard format response message. The local statistics method corresponds to compensatory metering for missing data on the channel side, thus avoiding billing gaps when some channels do not return the usage field under streaming output or private protocols.

[0046] The billing processing unit calculates fees based on the input unit price ratio, output unit price ratio, and user group ratio of the user in the relay context corresponding to the target model. If the actual fee is denoted as C, the input token quantity as N1, the output token quantity as N2, the input unit price ratio as R1, the output unit price ratio as R2, and the user group ratio as R3, then C = (N1 × R1 + N2 × R2) × R3. Here, N1 is determined by the token statistics of the request body input content, N2 is determined by the token statistics of the response output content, R1 and R2 come from the billing configuration corresponding to the target model in the pricing table, and R3 is used to reflect the rate differences at the billing level for different user groups. The same target model can yield consistent fee results across different channels, different user groups, and different output scales.

[0047] The account processing unit uses the call serial number as an idempotent verification key to perform atomic balance deduction on the account balance data table, ensuring consistency between balance changes, fee settlement results, and call records. When the balance is insufficient or the transaction fails, a billing exception status is output, and balance update confirmation is stopped. After successful balance deduction, the log processing unit writes the call timestamp, user identifier, target channel identifier, model name, token consumption, and fee details to the log table, simultaneously recording audit fields such as the response status code, request delay, and call serial number. Once the balance deduction confirmation and log writing are complete, a standard format response message with the replenished usage information and completed fee settlement is returned to the request initiator.

[0048] The cross-modal AI processing method in the embodiments of the present invention has been described above. The cross-modal AI platform in the embodiments of the present invention is described below. Please refer to [link / reference]. Figure 5 One embodiment of the cross-modal AI platform in this invention includes: Analysis module 501 is used to generate a relay context carrying a set of modality types and the target model name based on the platform API key in the request header, and to determine the target channel configuration information and alternative channel identifiers from the registered channels in the channel capability table. The remapping module 502 is used to perform structured remapping of message list fields, removal of unsupported parameters and injection of channel API keys on the request body based on the target channel configuration information, so as to obtain the target channel format request message and send the target channel format request message to the service endpoint in the target channel configuration information to obtain the downstream channel response message. The standardization processing module 503 is used to perform reverse structured remapping and streaming push data frame standardization processing on the downstream channel response message to obtain a standard format response message and return it to the request initiator.

[0049] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0050] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0051] The above-described embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A cross-modal AI processing method, characterized in that, include: Generate a relay context carrying a set of modality types and the target model name based on the platform API key in the request header, and determine the target channel configuration information and alternative channel identifiers from the registered channels in the channel capability table; Based on the target channel configuration information, the message list field structure remapping, non-supported parameter removal and channel API key injection processing are performed on the request body to obtain the target channel format request message, and the target channel format request message is sent to the service endpoint in the target channel configuration information to obtain the downstream channel response message; The downstream channel response message is subjected to reverse structured remapping and streaming push data frame standardization processing to obtain a standard format response message and return it to the request initiator.

2. The cross-modal AI processing method according to claim 1, characterized in that, The process of generating a relay context carrying a set of modality types and the target model name based on the platform API key in the request header, and determining the target channel configuration information and alternative channel identifiers from the registered channels in the channel capability table, includes: Perform a key-value matching query on the platform API key in the request header to obtain the token permission configuration; Based on the whitelist of allowed call models in the token permission configuration, the modal tag set of the request body and the target model name in the request body are written into the relay context. The modal tag set includes text modality, image modality and audio modality. Based on the relay context, the target channel configuration information and alternative channel identifiers are determined from the registered channels in the channel capability table.

3. The cross-modal AI processing method according to claim 2, characterized in that, The step of determining the target channel configuration information and alternative channel identifiers from the registered channels in the channel capability table based on the relay context includes: Match the registration channels in the channel capability table that have a mapping record with the target model name in the relay context and are in the enabled state, and verify whether the registration channel capability tag field covers all modal types in the modal type set. The registration channels that pass the verification are determined as the candidate channel set. The target score of each candidate channel is calculated based on the unit price ratio of the pricing table model corresponding to each candidate channel in the candidate channel set, the recent average response latency in the request success rate statistics queue, and the remaining quota of the channel. The configuration information of the candidate channel with the highest target score is used as the target channel configuration information, and the candidate channel with the second highest target score is used as the alternative channel identifier.

4. The cross-modal AI processing method according to claim 3, characterized in that, Based on the target channel configuration information, the request body undergoes structured remapping of message list fields, removal of unsupported parameters, and injection of channel API keys to obtain a target channel format request message. This target channel format request message is then sent to the service endpoint specified in the target channel configuration information to obtain a downstream channel response message, including: Based on the channel type identifier in the target channel configuration information, the corresponding adapter plugin is loaded from the adapter registry, a structured remapping is performed on the message list fields of the request body, and the non-supported parameter fields in the request body are removed according to the adapter plugin to obtain the remapped request body; Based on the channel authentication mechanism type in the target channel configuration information, a channel API key is injected into the remapping request body to obtain a target channel format request message. The target channel format request message is then sent to the service endpoint in the target channel configuration information to obtain a downstream channel response message.

5. The cross-modal AI processing method according to claim 4, characterized in that, The process involves loading the corresponding adapter plugin from the adapter registry based on the channel type identifier in the target channel configuration information, performing structured remapping on the message list fields of the request body, and removing unsupported parameter fields from the request body according to the adapter plugin, to obtain a remapped request body, including: The corresponding adapter plugin is loaded from the adapter registry based on the channel type identifier in the target channel configuration information; Perform a role-by-role traversal and structured remapping on the message list fields of the request body, and perform a key-by-key comparison of all parameter field keys in the request body with the parameter support whitelist maintained by the adapter plugin. Remove parameter fields whose keys do not exist in the parameter support whitelist to obtain the remapped request body.

6. The cross-modal AI processing method according to claim 5, characterized in that, The step involves injecting a channel API key into the remapping request body based on the channel authentication mechanism type in the target channel configuration information, obtaining a target channel format request message, and sending the target channel format request message to the service endpoint in the target channel configuration information to obtain a downstream channel response message, including: Obtain the channel authentication mechanism type from the target channel configuration information; When the channel authentication mechanism type is an authorization header type, the channel API key is injected into the authorization field of the request header to obtain the target channel format request message; when the channel authentication mechanism type is a query parameter type, the channel API key is concatenated to the query parameter of the server endpoint address to obtain the target channel format request message; when the channel authentication mechanism type is a custom header type, the authentication header is constructed according to the target channel's private authentication format to obtain the target channel format request message. The target channel format request message is sent to the server endpoint, and a downstream channel response message is returned.

7. The cross-modal AI processing method according to claim 6, characterized in that, If the status code of the returned downstream channel response message is a channel-side service exception type, then the alternative channel configuration information is read from the channel capability table using the alternative channel identifier, the channel API key injection process is re-executed on the remapping request body, and the message is resent to the alternative channel service endpoint to obtain the downstream channel response message.

8. The cross-modal AI processing method according to claim 1, characterized in that, The step of performing reverse structured remapping and streaming push data frame standardization on the downstream channel response message to obtain a standard format response message and returning it to the request initiator includes: Perform reverse structured remapping on the private fields carrying the model output content in the response body of the downstream channel response message, write the downstream channel push data stream of the streaming response into a standard format push data frame, and push the standard format push data frame to the request initiator. For each standard format push data frame in the downstream channel push data stream, a stream end signal detection is performed. When a private stream end marker of the downstream channel is detected, a stream termination marker is pushed to the request initiator to obtain a standard format response message, and the standard format response message is returned to the request initiator.

9. The cross-modal AI processing method according to claim 8, characterized in that, If the token consumption is missing in the usage field of the standard format response message, the token number is counted locally by the token segmenter for the input content of the request body and the output content of the response, and the statistical result is used to fill the token consumption. Based on the Token consumption, the input unit price ratio and output unit price ratio of the corresponding target model in the pricing table, and the user group ratio to which the user belongs in the relay context, the fee calculation is performed. An atomic balance deduction is performed on the account balance data table, and the call timestamp, user identifier, target channel identifier, model name, Token consumption, and fee details are written to the log table. The standard format response message is then returned to the request initiator.

10. A cross-modal AI platform, characterized in that, A method for performing cross-modal AI processing as described in any one of claims 1-9, comprising: The analysis module is used to generate a relay context carrying a set of modality types and the target model name based on the platform API key in the request header, and to determine the target channel configuration information and alternative channel identifiers from the registered channels in the channel capability table; The remapping module is used to perform structured remapping of message list fields, removal of unsupported parameters, and injection of channel API keys on the request body based on the target channel configuration information, so as to obtain a target channel format request message, and send the target channel format request message to the service endpoint in the target channel configuration information to obtain a downstream channel response message. The standardization processing module is used to perform reverse structured remapping and streaming push data frame standardization processing on the downstream channel response message to obtain a standard format response message and return it to the request initiator.

Images