Positioning request processing method and positioning request processing system

Abstract

Embodiments of the present specification provide a positioning request processing method and a positioning request processing system, wherein the positioning request processing method comprises: obtaining target multi-modal data and a positioning request for a target object in the target multi-modal data; performing visual processing on the target multi-modal data to obtain at least one visual data, wherein the visual data carries evidence content of the target object; rendering a time coordinate to the visual data to obtain multi-modal data with a time coordinate, wherein the time coordinate is located in the same visual data as the evidence content; inputting the multi-modal data with the time coordinate and the positioning request into a preset multi-modal data processing model, performing inference on the multi-modal data with the time coordinate based on the positioning request, and obtaining a corresponding time interval of the target object in the target multi-modal data. The overall processing does not involve adjustment of model parameters, widens the processing application range, and improves the accuracy of determining the corresponding time interval of the target object.

Description

Technical Field

[0001] The embodiments in this specification relate to the field of multimodal data processing technology, and in particular to a location request processing method and a location request processing system. Background Technology

[0002] With the rapid development of multimodal large language model technology, time-based localization of target objects in multimodal data such as audio and video has become an important capability supporting refined content processing in fields such as long video retrieval, content review, and meeting quality inspection.

[0003] Currently, for time localization of target objects in multimodal data, a time-aware training approach can usually be adopted. This involves introducing a timestamp-aware encoder or dedicated time terms into the model structure and training the model using time-labeled samples, enabling the model to output the time interval corresponding to the target object. Alternatively, there is also a approach that explicitly adds time display information to the model input and combines it with a query-driven model for target object time localization.

[0004] However, the aforementioned technical solutions have certain technical problems in practical applications. The training-based approach requires modification of the model structure and relies on retraining with a large number of time-labeled samples, making it difficult to apply to models with fixed parameters or those only providing an inference interface. In the approach of explicitly adding time display information, the added time information is usually independent of the evidence content of the target object in the media, making it difficult for the model to correlate the time information with the evidence content, resulting in low accuracy in determining the corresponding time interval of the target object in the target multimodal data. Therefore, there is an urgent need for a location request processing method that does not require updating model parameters and provides more accurate time interval positioning. Summary of the Invention

[0005] In view of the above, embodiments of this specification provide a location request processing method. One or more embodiments of this specification also relate to a location request processing system, a computing device, a computer-readable storage medium, and a computer program product, to solve the aforementioned technical problems.

[0006] According to a first aspect of the embodiments of this specification, a location request processing method is provided, including: Acquire target multimodal data, and request the location of target objects within the target multimodal data; Visualize the target multimodal data to obtain at least one visual data, wherein the visual data carries the evidentiary content of the target object; Render the time coordinates onto the visual data to obtain multimodal data with time coordinates, where the time coordinates and the evidence content are located in the same visual data. The multimodal data with time coordinates and the positioning request are input into a preset multimodal data processing model. Based on the positioning request, the multimodal data with time coordinates is inferred to obtain the time interval corresponding to the target object in the target multimodal data.

[0007] According to a second aspect of the embodiments of this specification, a location request processing system is provided, including a front-end interface and a location request processing module; The front-end interface is used to receive target multimodal data and location requests for target objects in the target multimodal data, and send the target multimodal data and location requests to the location request processing module. The location request processing module is used to obtain target multimodal data and location requests, and execute the location request processing method as described above to obtain the time interval corresponding to the target object in the target multimodal data, and return it to the front-end interface.

[0008] According to a third aspect of the embodiments of this specification, a computing device is provided, comprising: Memory and processor; The memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions, which, when executed by the processor, implement the steps of the above-described location request processing method.

[0009] According to a fourth aspect of the embodiments of this specification, a computer-readable storage medium is provided that stores computer-executable instructions, which, when executed by a processor, implement the steps of the location request processing method described above.

[0010] According to a fifth aspect of the embodiments of this specification, a computer program product is provided, including computer-executable instructions that, when executed by a processor, implement the steps of the location request processing method described above.

[0011] One embodiment of this specification implements a location request processing method, which visualizes the target multimodal data, renders the time coordinates onto the visual data carrying the evidence content of the target object, and infers the multimodal data with time coordinates and the location request input multimodal data processing model to determine the time interval corresponding to the target object in the target multimodal data.

[0012] By acquiring target multimodal data and requesting the location of target objects within that data, a clear input foundation is provided for time-based location of target objects. Visualizing the target multimodal data yields visual data carrying the evidence content of the target objects, presenting the evidence content in a unified visual format and providing a suitable visual carrier for subsequent time information injection. Rendering time coordinates onto the visual data and placing them within the same visual data set as the evidence content achieves a structural binding between the time coordinates and the evidence content of the target objects on the model input side, allowing the model to directly... Establishing a correspondence between time coordinates and evidence content reduces the reliance on time representation learning during the training phase. By inferring between multimodal data with time coordinates and the multimodal data processing model that inputs the location request, the time interval corresponding to the target object in the target multimodal data is obtained. This fully utilizes the model's inference ability for multimodal inputs to complete time localization. The entire process does not involve adjusting model parameters and can be applied to already trained models, thus broadening the applicability of time localization schemes. At the same time, since the time coordinates are visiblely attached to the visual data where the evidence content is located, the accuracy of determining the time interval corresponding to the target object is improved. Attached Figure Description

[0013] Figure 1 This is a flowchart of a location request processing method provided in one embodiment of this specification; Figure 2 This is a schematic diagram of the framework of a location request processing method provided in one embodiment of this specification; Figure 3 This is a schematic diagram of the structure of a location request processing system provided in one embodiment of this specification; Figure 4 This is a structural block diagram of a computing device provided in one embodiment of this specification. Detailed Implementation

[0014] Many specific details are set forth in the following description to provide a full understanding of this specification. However, this specification can be implemented in many other ways than those described herein, and those skilled in the art can make similar extensions without departing from the spirit of this specification. Therefore, this specification is not limited to the specific implementations disclosed below.

[0015] The terminology used in one or more embodiments of this specification is for the purpose of describing particular embodiments only and is not intended to be limiting of the one or more embodiments of this specification. The singular forms “a,” “described,” and “the” as used in one or more embodiments of this specification and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used in one or more embodiments of this specification refers to and includes any or all possible combinations of one or more associated listed items.

[0016] It should be understood that although the terms first, second, etc., may be used to describe various information in one or more embodiments of this specification, such information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, first may also be referred to as second without departing from the scope of one or more embodiments of this specification, and similarly, second may also be referred to as first. Depending on the context, the word "if" as used herein may be interpreted as "when," "when," or "in response to a determination."

[0017] Furthermore, it should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in one or more embodiments of this specification are obtained through open-source datasets or public datasets that comply with their license agreements, or are obtained with full authorization from the relevant parties. Moreover, the collection, use and processing of the relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and corresponding operation entry points are provided for users to choose to authorize or refuse.

[0018] In one or more embodiments of this specification, a large model is a deep learning model with a large number of model parameters, typically containing hundreds of millions, tens of billions, hundreds of billions, trillions, or even tens of trillions of model parameters. A large model can also be called a foundation model. It is pre-trained using large-scale unlabeled corpora to produce a pre-trained model with hundreds of millions of parameters. Such models can adapt to a wide range of downstream tasks and have good generalization ability. Examples include Large Language Models (LLMs) and multi-modal pre-training models.

[0019] In practical applications, large models only require a small number of samples to fine-tune the pre-trained model before they can be applied to different tasks. Large models can be widely used in fields such as Natural Language Processing (NLP) and Computer Vision. Specifically, they can be applied to computer vision tasks such as Visual Question Answering (VQA), Image Captioning (IC), and Image Generation, as well as NLP tasks such as text-based sentiment classification, text summarization, and machine translation. The main application scenarios for large models include digital assistants, intelligent robots, search, online education, office software, e-commerce, and intelligent design.

[0020] First, the terms and concepts used in one or more embodiments of this specification will be explained.

[0021] Large Language Models (LLMs) are large-scale language models that are pre-trained on massive amounts of text data. They typically have a large number of model parameters and possess strong natural language understanding and generation capabilities.

[0022] Autoregressive is a sequence generation method that generates output units sequentially and uses the generated parts as conditions for subsequent generation.

[0023] Pre-training is a training method that involves training a model in advance on large-scale general data to obtain general capabilities. The model obtained through pre-training can usually be adapted and transferred to a variety of downstream tasks with a small amount of data.

[0024] Forward inference is the process of calculating the input data layer by layer according to the model's predetermined structure and parameters to obtain the output. This process does not involve updating the model parameters.

[0025] The Short-Time Fourier Transform (STFT) is a method of processing a signal by dividing it into short time windows and then performing a Fourier transform on each segment to obtain the signal spectrum as it changes over time.

[0026] The Mel scale is a frequency scale designed to simulate the non-linear perception of sound frequencies by the human ear, used to map linear frequencies to Mel frequencies.

[0027] A Log-Mel spectrogram is a two-dimensional time-frequency graph that uses the Mel scale as the frequency axis and takes the logarithm of the energy. It is used to present the distribution of the energy of a sound signal over time and frequency.

[0028] The Intersection over Union (IoU) ratio is a measure of the degree of overlap between two intervals or regions. It is the ratio of the intersection to the union of the two intervals, and the value is between zero and one. The larger the value, the higher the degree of overlap.

[0029] The coefficient of determination is a statistic used in regression prediction to measure the extent to which the prediction results explain the target variable. Its value is usually no more than one, and the closer it is to one, the better the prediction results explain the changes in the target variable.

[0030] A hook mechanism is a mechanism that registers callback functions at specified locations in the execution flow of a program or model. These functions are automatically triggered when the program passes through that location and are used to read or modify intermediate data.

[0031] Middleware is a software layer located between upper-layer applications and lower-layer systems, designed to provide general capabilities. It provides standardized interfaces to upper-layer applications and transparently enhances the capabilities of the lower-layer systems.

[0032] An inference engine is a software or service component specifically designed to perform model inference computations, enabling efficient scheduling of the model's forward computation process.

[0033] With the rapid development of multimodal large language model technology, the temporal localization of target objects in multimodal data such as audio and video has become a common processing requirement in scenarios such as long video retrieval, content review, and meeting quality inspection.

[0034] Currently, time-based localization can be achieved through a time-aware training approach, which involves modifying the model structure, introducing dedicated time terms, and training the model using time-labeled samples. Alternatively, time information can be explicitly added to the model input, and localization can be achieved by combining a query-driven model targeting the target object.

[0035] However, the above-mentioned solutions have certain technical limitations in practical applications. Training-based solutions are constrained by modifications to the model structure and reliance on time-labeled samples, making them difficult to apply directly in scenarios where model parameters are fixed or only an inference interface is provided. In solutions that explicitly add time display information, the time information is usually presented independently of the evidence content, making it difficult for the model to correlate the two, resulting in limited accuracy in time interval localization. Therefore, there is an urgent need for a localization request processing method that can improve the accuracy of time interval localization while keeping model parameters unchanged.

[0036] In view of this, this specification provides a location request processing method, and also relates to a location request processing system, a computing device, a computer-readable storage medium, and a computer program product, which will be described in detail in the following embodiments.

[0037] Specifically, see Figure 1 , Figure 1 A flowchart of a location request processing method according to an embodiment of this specification is shown, such as... Figure 1 As shown, the method specifically includes the following steps.

[0038] Step 102: Obtain target multimodal data and a location request for the target object in the target multimodal data.

[0039] The location request processing method provided in the embodiments of this specification can be applied to various scenarios that require time positioning of target objects in multimodal data such as audio and video. Examples include long video and audio retrieval and editing, video content review and evidence collection, smart cockpit and security event positioning, and meeting recording and customer service recording quality inspection, where it is necessary to determine the start and end times of events or objects appearing in the media. It can be deployed and run in system environments such as multimedia retrieval platforms, content review and evidence collection systems, recording quality inspection systems, and multimodal inference service capability enhancement middleware, providing processing support for the need to perform accurate time positioning of multimodal data according to natural language queries.

[0040] Target multimodal data refers to multimodal data to be processed for temporal localization, i.e., data containing the target object to be located and whose corresponding time interval needs to be determined. Specifically, target multimodal data can include at least one of audio data and video data; if the target multimodal data is audio data, it can be a meeting recording to be quality inspected, a long audio program to be retrieved, or an audio stream containing acoustic events; if the target multimodal data is video data, it can be a surveillance video, screen recording content, or a video clip containing visual events. Furthermore, target multimodal data can be pre-stored in the form of media files or acquired in real time by acquisition devices; the target multimodal data can contain one or more target objects, and different target objects may correspond to different time intervals in the target multimodal data. The time interval is used to characterize the time range in which the target object appears or occurs in the target multimodal data.

[0041] The target object is the object that the location request points to and that needs to be located temporally within the target multimodal data. Specifically, the target object can be an acoustic event, a visual event, or a specific content fragment appearing in the target multimodal data. For example, the target object can be an acoustic event such as a gunshot, a siren, broken glass, or specific dialogue in audio; it can be a visual event such as a streetlight turning on, a vehicle passing by, or the appearance of a specific object in video; or it can be semantic content such as dialogue fragments involving refunds in audio and video. Furthermore, the target object is usually described in natural language by the location request; the target object has corresponding evidential content and a time interval in the target multimodal data. The evidential content is the specific presentation of the target object in the target multimodal data, and the time interval is the range of time within which the target object appears or occurs in the target multimodal data.

[0042] A location request is a request message used to instruct the temporal location of a target object in target multimodal data. The location request carries descriptive information about the target object. Specifically, in one optional embodiment, the location request is expressed in natural language, such as a question like "At what second did the sound of breaking glass occur?" or "At what second did the streetlights turn on?". In another optional embodiment, the location request is expressed in a structured form, which may include a target object identifier and a location instruction field. Further, the location request can be associated with the target multimodal data and provided to the processing end executing the location request processing method. The target object described in the location request corresponds to the actual content contained in the target multimodal data, so that subsequent processing can determine the time interval corresponding to the target object in the target multimodal data according to the semantics of the location request.

[0043] In practical applications, target multimodal data can be obtained through various methods.

[0044] One alternative method is to read pre-stored target multimodal data files from local or shared storage. Specifically, the media file corresponding to the target multimodal data can be located according to a preset reading path, and the media file can be decoded to obtain media data suitable for subsequent visualization processing. The encapsulation and encoding format of the media file can be determined based on the actual storage format; for example, common audio and video encapsulation formats can be used. Obtaining target multimodal data by reading stored media files supports offline time-based processing of historical media, facilitating batch processing of existing media in scenarios such as retrieval and quality inspection.

[0045] Another alternative approach is to acquire target multimodal data in real time using acquisition devices. Specifically, ambient audio can be acquired in real time using microphones, microphone arrays, or recording devices, and scene video can be acquired in real time using cameras or recording devices. The acquired data can then be used as target multimodal data for subsequent processing. Real-time acquisition of target multimodal data supports time-based localization processing of real-time audio and video streams, making it suitable for scenarios with high timeliness requirements, such as smart cockpits and security monitoring.

[0046] Another alternative approach is to receive target multimodal data pushed by upstream systems or platforms via an application programming interface (API). Specifically, after obtaining the media to be processed, the recording quality inspection platform, content review platform, or media management platform can send the multimodal data to be processed as target multimodal data to the processing end by calling the API provided by the location processing end. Receiving the data through the API allows the location request processing method to be embedded as a capability component into the upstream task flow, expanding the application scope of time-based location processing.

[0047] In addition, the acquired target multimodal data can be preprocessed. For example, the sampling format, channel form, frame rate or duration of the target multimodal data can be unified. The target multimodal data can also be segmented, denoised or truncated to make the target multimodal data meet the input requirements of subsequent visualization and inference processing.

[0048] In practical applications, obtaining a location request for a target object in multimodal data can also be achieved through various methods.

[0049] One alternative approach is to receive natural language location requests input by the user through a front-end interactive interface. Specifically, the front-end interactive interface may include a query input box or dialog window, where the user can enter a natural language query describing the target object, such as "find the clip of a car horn honking in a video." The processing end will then treat this natural language query as a location request. Simultaneously with entering the location request, the user can also select or upload target multimodal data associated with that request. Receiving location requests through the front-end interactive interface allows for direct acquisition of the user's location intent, facilitating the provision of interactive time-based location services.

[0050] Another alternative approach is to receive location requests from upstream task systems via an application programming interface (API). Specifically, retrieval platforms, review systems, or quality inspection systems can generate location requirements for the target object within the task flow, and then send the location request along with the target's multimodal data to the processing end by calling the API. Receiving location requests through the API allows the location request processing methods to be integrated and called by other systems, meeting the needs of automated task scenarios.

[0051] Another option is to generate location requests based on preset location task templates. Specifically, in content review scenarios, location requests for corresponding events can be automatically generated based on templates corresponding to the violation type. For example, a "dangerous event type" template can be used to automatically generate location requests for events such as horn honking or collisions. In quality inspection scenarios, location requests for key phrases or controversial segments can be automatically generated based on quality inspection rule templates. By generating location requests using templates, time-based location tasks can be initiated in batches without manual input.

[0052] In addition, location requests can be retrieved from historical location request records. For example, pending location request records can be read from logs or task lists to support the reprocessing of historical tasks or retrying of failures.

[0053] In this step, by acquiring the target multimodal data and the location request for the target object in the target multimodal data, an interrelated input basis is provided for subsequent visualization processing, temporal coordinate rendering, and inference processing. This enables subsequent processing to revolve around the temporal location requirements of the target object in the target multimodal data, avoiding the computational resource overhead caused by indiscriminately processing the entire media segment. At the same time, it enables temporal location processing to be performed based on the semantics described in the location request, providing a prerequisite for determining the accuracy of the time interval corresponding to the target object.

[0054] Step 104: Visualize the target multimodal data to obtain at least one visual data, wherein the visual data carries the evidence content of the target object.

[0055] Visualization processing is the process of converting target multimodal data into visual data form, providing a visual carrier that can bear temporal coordinates for the temporal localization of the target object. Specifically, visualization processing can be performed separately on the audio and video components of the target multimodal data: for the audio component, it can be converted into a two-dimensional image visual representation, presenting acoustic information as visual patterns; for the video component, image frames can be extracted from the video data, presenting the video footage as an image sequence. Furthermore, visualization processing is completed before inference, does not require the introduction of learnable processing parameters, and the visualized visual data maintains a temporal correspondence with the target multimodal data, meaning that each position in the visual data corresponds to each moment in the target multimodal data.

[0056] Visual data is image-based data that carries the evidentiary content of the target object, obtained after visualizing the target multimodal data. Specifically, in one optional embodiment, there is one set of visual data; in another optional embodiment, there are multiple sets of visual data. The visual data can present the acoustic or visual evidentiary content of the target object in the target multimodal data. For example, if the target object is an acoustic event, the visual data can include the sound energy pattern corresponding to the acoustic event, such as a spectrum diagram; if the target object is a visual event, the visual data can include the visual content of the visual event. Furthermore, the visual data simultaneously serves two functions: firstly, it acts as a carrier for presenting the evidentiary content of the target object, enabling the model to identify the evidentiary content of the target object from the visual data; secondly, it acts as a carrier of time coordinates, ensuring that the rendered time coordinates are located in the same visual data as the evidentiary content, thereby supporting the model in establishing the correspondence between the time coordinates and the evidentiary content.

[0057] Evidence content refers to the specific presentation of the target object within the target multimodal data; that is, the acoustic or visual signal content upon which the model relies to recognize the target object. Specifically, when the target object is an acoustic event, the evidence content can be the corresponding sound signal segment in the audio, such as the waveform or energy distribution segment corresponding to the sound of breaking glass. When the target object is a visual event, the evidence content can be the corresponding scene content in the video, such as the area of ​​light change in the scene corresponding to a streetlight turning on. Furthermore, the evidence content is usually distributed over a continuous time period within the target multimodal data, and the range corresponding to this time period is the time interval corresponding to the target object. By embedding the evidence content in the visual data, the existence of the target object and its temporal location can be perceived by the model simultaneously at the visual level, providing a basis for binding time coordinates with the evidence content.

[0058] In practical applications, visualizing target multimodal data to obtain at least one visual data point can be achieved through various methods.

[0059] One alternative approach is to perform a time-frequency transformation on the audio portion of the target multimodal data to obtain a two-dimensional time-frequency image reflecting the change of sound energy over time, which can then be used as visual data. Specifically, the audio signal can be first divided into frames and windowed according to a preset window. Fourier transforms can be performed on each frame to obtain the spectrum. Then, the frequency axis is mapped according to human auditory perception characteristics and logarithmically compressed. The spectra of each frame are arranged in chronological order to form a two-dimensional image where the horizontal axis represents time, the vertical axis represents frequency, and color intensity represents energy strength. This image can carry the evidentiary content of the acoustic events in the audio. By obtaining visual data through time-frequency transformation, one-dimensional acoustic evidence can be converted into a two-dimensional image, enabling multimodal data processing models that use images as input to perceive the acoustic evidence content and its temporal distribution.

[0060] Another alternative approach is to extract image frame sequences from the video portion of the target multimodal data, using each image frame as visual data. Specifically, image frames can be extracted from the video data at a preset sampling frequency or interval, or the video data can be extracted by keyframes to obtain an image frame sequence that is temporally aligned with the video data. Each image frame carries the video content at the corresponding moment, serving as a visual carrier of the target object's evidentiary content in the video. By obtaining visual data through frame extraction, the visual evidence of the target object can be directly utilized from the video frame, preserving spatial detail information within the image.

[0061] Another alternative approach is to jointly visualize the audio and video components of the target multimodal data, organizing the visual representations corresponding to the audio and video frames in a time-aligned manner to form a visual data set. Specifically, while extracting video frames, a time-frequency transformation can be performed on the audio component, and the two types of visual data can be aligned and arranged in chronological order to obtain a visual data set that includes both acoustic and visual evidence. Through joint visualization, the evidence content of different modalities in the multimodal data can be preserved, providing a more complete visual input for subsequent temporal localization of the multimodal evidence content.

[0062] In addition, visual data can be generated by first unifying the duration and then cropping it according to the content of the evidence of the target object. For example, the multimodal data of the target can be segmented first, and only the segments containing the content of the candidate evidence can be visualized to reduce the amount of visual data processing.

[0063] In this step, by converting the information carrying the evidence content of the target object in the target multimodal data into at least one visual data, the subsequent processing can directly carry the evidence content of the target object in the form of an image. At the same time, it provides the visual carrier required for binding the time coordinates, so that the association between the time coordinates and the evidence content no longer depends on the implicit alignment capability inside the model, thus providing a processing foundation for improving the accuracy of time positioning.

[0064] Step 106: Render the time coordinates to the visual data to obtain multimodal data with time coordinates, where the time coordinates and evidence content are located in the same visual data.

[0065] Time coordinates are coordinate information used to characterize the temporal position of a target object within target multimodal data. Specifically, time coordinates can be presented in the form of visible graphics, such as time scales, time markers, or other visual forms. The time position represented by the time coordinates corresponds to the actual time in the target multimodal data, and the media time corresponding to each position in the visual data can be determined by reading the time coordinates. Furthermore, time coordinates are information explicitly injected into the model input from the outside, unlike the implicit time representation internalized by the model during the training phase. Visual data with rendered time coordinates allows the model to directly read media time information from the input without relying on the time representation capabilities learned during the training phase, thus enabling time localization processing to be applied to any trained model.

[0066] Multimodal data with time coordinates is multimodal data rendered with time coordinates added; it serves as the multimodal input for multimodal data processing models to perform inference. Specifically, multimodal data with time coordinates can be constructed based on rendered visual data and other components of the target multimodal data. In this data, the time coordinates exist as visible markers within the visual data, which also carries evidence of the target object. Furthermore, multimodal data with time coordinates, together with the location request, constitutes the model's input. Based on the correspondence between the visible time coordinates and the evidence content in the multimodal data with time coordinates, the model can determine the time interval corresponding to the target object.

[0067] In practical applications, rendering time coordinates to visual data to obtain multimodal data with time coordinates can be achieved through various methods.

[0068] One alternative approach is to directly plot the time coordinates as visual markers within the content area of ​​the visual data, integrating the time coordinate markers with the visual data carrying the evidence. Specifically, the location for presenting the time coordinate markers can be determined within the visual data, and a visible time marker that changes with time position can be plotted at that location, for example, plotting the corresponding time marker at the media moment corresponding to each visual data point. The plotting process is performed directly on the visual data carrying the evidence, without generating a separate, independent frame containing the time coordinates. By directly plotting the time coordinates on the visual data carrying the evidence, the time coordinates and the evidence content are located within the same visual data, allowing the model to read the corresponding time coordinates while perceiving the evidence content.

[0069] Another alternative approach is to label different regions in the visual data with corresponding time information based on the correspondence between content and time, forming visual data that binds regions to time. Specifically, based on the time alignment established during visualization processing, the media time corresponding to each content position in the visual data can be determined, and the time information can be labeled onto the visual data according to this correspondence, so that the region in the visual data carrying the evidence content corresponds to its time of occurrence on the image. Through region-level time labeling, time coordinates can be precisely bound to the image region where the evidence content is located, strengthening the correspondence between time coordinates and evidence content.

[0070] Another alternative approach is to draw a time scale along the edges or sidebands of the visual data, keeping the scale and the main body of the visual data within the same image object. Specifically, the time scale can be drawn at the bottom or top edge of the visual data, with the scale's position corresponding to the media's time point. The evidence content in the visual data and the scale belong to the same image object, and the time corresponding to the evidence content can be directly read from the scale. By drawing the time scale within the same image object, the model can simultaneously perceive the evidence content and the time coordinates without obscuring the evidence content.

[0071] In addition, a rendering method that preserves the integrity of the evidence content while rendering the time coordinates can be adopted. For example, time coordinate markers can be drawn in a semi-transparent manner, or time coordinate markers can be drawn only in the preset corner areas to avoid the time coordinates covering the evidence content and affecting the model's recognition of the evidence content.

[0072] In this step, by rendering the time coordinates to the visual data and constraining the time coordinates and evidence content to be located in the same visual data, the time coordinates and evidence content are structurally bound on the input side. The model can directly establish the correspondence between the evidence content and its time position, reducing the implicit reasoning burden required for the model to infer the time position from the evidence content. This provides a readable time basis for the multimodal data processing model to determine the time interval corresponding to the target object without updating the parameters.

[0073] Step 108: Input the multimodal data with time coordinates and the positioning request into the preset multimodal data processing model, and perform inference on the multimodal data with time coordinates based on the positioning request to obtain the time interval corresponding to the target object in the target multimodal data.

[0074] A multimodal data processing model is used to process multimodal data with time coordinates and location requests to determine the time interval corresponding to a target object. This model can handle inputs from multiple modalities, including visual data. Specifically, it can be a multimodal large language model, i.e., a large-scale language model capable of uniformly processing multiple modal inputs such as text, audio, and video. This model can be obtained through pre-training on large-scale multimodal data and can understand the semantic content of visual data and the semantics of natural language descriptions. Furthermore, the processing procedures in the embodiments of this specification are based on a pre-trained multimodal data processing model, and the model's parameters are not changed during the processing. The model can use an autoregressive approach to generate processing results for the input, and the processing results can be in the form of natural language text.

[0075] A time interval is the range of time during which a target object appears or occurs in the target multimodal data. Specifically, in one optional embodiment, the time interval is represented by a start time and an end time, for example, the time interval can be a pair of start and end times in seconds; in another optional embodiment, the time interval is represented by a normalized position, for example, it can be represented by a relative position interval within the media duration. Further, the time interval is the final positioning result output by the positioning request processing method. The closer the time interval is to the actual occurrence range of the target object in the target multimodal data, the more accurate the time positioning. After determining the time interval, downstream processing such as slicing, retrieval, tagging, or review of the target multimodal data can be performed based on the time interval.

[0076] In practical applications, inputting multimodal data with time coordinates and positioning requests into a preset multimodal data processing model can be achieved through various methods.

[0077] One alternative approach is to assemble the time-coordinated multimodal data and localization requests into the model according to the input interface requirements of the multimodal data processing model. Specifically, the model's input interfaces for visual, audio, and text modalities can be obtained separately. The visual portion of the time-coordinated multimodal data is mapped to the visual input interface, and the other modal components are mapped to their corresponding modal input interfaces. The localization request is then sent as text input to the text input interface, where it is uniformly encoded and processed by the model. By assembling the input according to the model's input interfaces, it is possible to adapt to trained models with different structures, enabling time-localization processing to be applied to models with fixed parameters or those that only provide an inference interface.

[0078] Another alternative approach is to combine time-coordinated multimodal data with location requests into a unified input format supported by the model before feeding it into the model. Specifically, when the model supports receiving multimodal content in a unified input sequence format, the time-coordinated multimodal data and location requests can be organized into an input sequence according to a preset order, and then fed into the model for encoding and inference. By feeding the model with a unified input format, the dependence of the input assembly process on the model structure can be reduced, facilitating rapid integration with existing inference services.

[0079] In addition, the size or length of multimodal data with time coordinates can be adapted before input. For example, the size of visual data can be standardized, the amount of visual data can be controlled, or non-critical content can be compressed to adapt to the input capacity limitations of the model.

[0080] After inputting the time-coordinated multimodal data and the positioning request into the preset multimodal data processing model, inference can be performed on the time-coordinated multimodal data based on the positioning request to obtain the time interval corresponding to the target object in the target multimodal data.

[0081] One alternative approach involves a multimodal data processing model performing forward inference on multimodal data with time coordinates. During inference, the model combines the semantic recognition of the target object with the location request to determine the evidence content, and uses the visible time coordinates in the visual data to determine the corresponding time position of the evidence content. Finally, the model outputs the time interval corresponding to the target object. Specifically, during forward inference, the model encodes the visual data, including both the semantic information of the evidence content and the location information corresponding to the visible time coordinates. The model combines the location request to locate the evidence content of the target object, reads its time coordinates to determine the corresponding start and end times, and writes these times into the output. The processing end extracts the start and end times from the model output as the time interval corresponding to the target object. By determining the time interval through forward inference and parsing the output, the model fully utilizes the content understanding capabilities of the trained model without requiring any parameter adjustments.

[0082] Another alternative approach is to determine the time interval corresponding to the target object through multiple inference voting. Specifically, media sampling and decoding parameters can be fixed, and multiple inferences can be performed on multimodal data with time coordinates and location requests to obtain multiple candidate time location results. These candidate results can then be statistically integrated; for example, the time interval corresponding to the highest frequency or the median result can be used as the time interval corresponding to the target object. Determining the time interval through multiple inference integrations can suppress random fluctuations in a single inference and improve the stability of the time interval.

[0083] Another option is to correct the time localization results obtained through inference by incorporating prior knowledge of the target object's category. Specifically, the duration or location distribution patterns corresponding to the target object's category can be known in advance. After the model obtains the initial time localization results, the results are fine-tuned according to the target object's category prior. For example, results that deviate significantly from the category prior duration are corrected to a reasonable range, and the corrected time interval is used as the time interval corresponding to the target object. By correcting with category priors, boundary jitter in the model's inference results can be reduced.

[0084] In addition, streaming or segmented processing methods can be used during the inference process. For example, when the target multimodal data is long, the multimodal data with time coordinates can be segmented for inference according to time windows, and the inference results of each segment can be integrated according to time coordinates to support time positioning in long media scenarios.

[0085] In this step, multimodal data with time coordinates and location requests are input into a multimodal data processing model for inference to obtain the time interval corresponding to the target object. This allows time-based location processing to be completed based on the model's direct understanding of the multimodal input. The processing does not involve adjusting model parameters and can be directly applied to models that have already been trained or only provide an inference interface, thus broadening the application scope of time-based location processing. At the same time, since the model can directly determine the time position corresponding to the evidence content based on the visible time coordinates in the input, the accuracy of time interval determination is improved, providing reliable location results for subsequent applications such as time interval-based retrieval, slicing, and review.

[0086] In the embodiments described in this specification, through the coordinated process of visualization processing, time coordinate rendering, and model inference, the evidence content of the target object and the visible time coordinates representing its time position are jointly contained in the same visual data. The time coordinates thus become a readable time reference on the input side, and time positioning no longer relies on the time representation internalized by the model during training. Based on this, the multimodal data with time coordinates and the multimodal data processing model for the location request input complete the inference. The entire process does not involve adjusting model parameters, thus it can be applied to models that have already been trained or only provide an inference interface, broadening the applicability of time positioning processing. Simultaneously, since the model can directly read the time position corresponding to the evidence content from the input, the accuracy of determining the corresponding time interval of the target object is improved, providing reliable positioning results for subsequent applications such as time interval-based retrieval, slicing, and review.

[0087] In one optional embodiment of this specification, the target multimodal data includes audio data, and the visual data includes a spectrogram; visual processing of the target multimodal data to obtain at least one visual data includes: Perform spectral conversion on the audio data to obtain a spectrogram; Rendering the time coordinates to visual data yields multimodal data with time coordinates, including: The time scale of the audio data is used as a time coordinate to render the spectrogram, resulting in a spectrogram with a time axis.

[0088] Audio data is the audio component of the target multimodal data; it is media data that carries sound signals and requires spectral conversion. Specifically, when the target multimodal data is pure audio media, the audio data can be the complete audio content of that media. When the target multimodal data contains both sound and visuals, in one optional embodiment, the audio data is the corresponding sound content. In another optional embodiment, the audio data is audio track data separated from the multimodal data. The audio data can be in the form of recordings, audio streams, etc. Furthermore, when the target object is an acoustic event appearing in the audio data, the evidence of the acoustic event exists within the audio data and needs to be presented in the form of a visual pattern after spectral conversion, thereby supporting the temporal localization processing of the acoustic event.

[0089] A spectrogram is a form of visual data, a two-dimensional graph obtained by spectral conversion of audio data, carrying evidence of acoustic events in the audio. Specifically, the spectrogram uses the horizontal axis to represent time, the vertical axis to represent frequency, and color or brightness to represent energy intensity. For example, the intensity of a color can represent the energy strength of a corresponding frequency component at a given moment. The sound signal corresponding to an acoustic event is presented as a specific energy pattern in the spectrogram, which is the visual representation of the acoustic event evidence. Furthermore, the spectrogram and the audio data maintain a temporal correspondence, with the horizontal axis position corresponding one-to-one with the moment in the audio data, providing a basis for rendering the time coordinates onto the spectrogram. In one optional embodiment, the visual data corresponding to the audio data is a single spectrogram; in another optional embodiment, the visual data corresponding to the audio data is multiple spectrograms.

[0090] A time scale is a graduation value in seconds that marks the time dimension of audio data, used to identify the time position corresponding to each moment in the audio data. Specifically, the time scale of audio data corresponds to the time base of the audio data itself. For example, the time dimension of audio data can be divided into units of seconds, so that each moment in the audio data corresponds to its own graduation value. Furthermore, the time scale is a specific representation of time coordinates. After rendering the time scale of audio data as time coordinates onto a spectrogram, the audio moment corresponding to each position in the spectrogram can be directly read from the time scale.

[0091] The time axis is a coordinate system on a spectrogram used to represent the time dimension. It is a visual element formed after the time scale of audio data is rendered onto the spectrogram. Specifically, tick lines are drawn along the horizontal axis of the spectrogram according to the time scale of the audio data, and the corresponding tick values, such as the corresponding second values, are marked on the tick lines to form the time axis of the spectrogram. The time axis and the energy pattern carrying the acoustic evidence content in the spectrogram are both located within the static image object of the spectrogram. Furthermore, the spectrogram with the time axis is a spectrogram with a time axis, which can be used as visual data input into a multimodal data processing model from multimodal data with time coordinates, allowing the model to determine the media moment corresponding to the acoustic evidence content based on the time axis.

[0092] In practical applications, audio data can be converted to a spectrum to obtain a spectrogram, which can be achieved through various methods.

[0093] One alternative approach is to render the waveform of the entire audio data segment as a log-Mel spectrogram, concentrating the acoustic information within a static visual object. Specifically, the audio data can be first divided into frames according to a preset window, and a short-time Fourier transform can be performed on each frame to obtain its spectrum. Then, the frequency axis is non-linearly mapped to a Mel scale according to human auditory perception characteristics, and the energy values ​​are logarithmically compressed. The processing results of each frame are arranged and rendered in chronological order to obtain the log-Mel spectrogram corresponding to the entire audio data segment, with the horizontal axis of the spectrogram corresponding to the time direction of the audio data. By rendering the entire segment to obtain the spectrogram, acoustic events are presented as continuous energy patterns within the same spectrogram, providing a time-consistent static visual carrier for rendering the time scale within the spectrogram.

[0094] Another alternative approach is to first segment the audio data, then perform spectral transformation on each segment to obtain a corresponding spectrogram. Specifically, the audio data can be divided into multiple segments according to a preset duration, and the temporal position of each segment within the audio data can be recorded. Then, spectral analysis can be performed on each segment to obtain its corresponding spectrogram. These spectrograms can be organized chronologically into a set or stitched together into a single complete spectrogram. Obtaining spectrograms through segmented transformation reduces the processing burden of single transformations of long audio data while preserving the temporal correspondence between each spectrogram and the audio data, facilitating consistent time reference when rendering time scales later.

[0095] Another option is to use a pre-defined spectrum conversion component to perform spectrum conversion on the audio data and obtain a spectrogram. Specifically, a processing component with audio spectrum analysis capabilities can be pre-configured. After the audio data is input into this component, it calculates the spectrum results corresponding to each time position. Alternatively, the component can directly output a spectrogram generated according to a set frequency scale. The output result is then processed with necessary scale mapping and energy compression to serve as the spectrogram. Obtaining the spectrogram by calling the spectrum conversion component allows for the reuse of relatively mature spectrum calculation capabilities, reduces the workload of implementing conversion logic independently, and facilitates integration with existing audio processing workflows.

[0096] In addition, audio data can be preprocessed before spectrum conversion. For example, the sampling rate and channel format of the audio data can be unified, and noise reduction, removal of silent segments, or amplitude normalization can be performed to make the obtained spectrogram more clearly present the evidence of acoustic events.

[0097] After obtaining the spectrogram, the time scale of the audio data needs to be rendered onto the spectrogram as a time coordinate, so that the time scale and the acoustic evidence content carried by the spectrogram are located in the same visual data. In practical applications, rendering the time scale of the audio data onto the spectrogram as a time coordinate to obtain a spectrogram with a time axis can also be achieved through various methods.

[0098] One alternative approach is to directly plot the time scale of the audio data along the horizontal axis of the spectrogram, creating a time axis within the spectrogram and obtaining a spectrogram with a time axis. Specifically, scale lines can be drawn along the horizontal axis of the spectrogram according to the time scale of the audio data, with corresponding scale values ​​labeled at the lines, such as the corresponding second values. The scale lines and values ​​together constitute the time axis of the spectrogram. The plotting process is completed within the static image object of the spectrogram, without generating a separate image to carry the time scale, nor presenting the time information as an overlay stream. By plotting the time scale directly on the spectrogram, the time coordinates and the evidence content are located in the same visual data, allowing the model to directly read the corresponding time position while recognizing the acoustic evidence content.

[0099] Another alternative approach is to annotate the time scale of the audio data onto the spectrogram based on the temporal correspondence between the spectrogram and the audio data, creating a spectrogram with a time axis. Specifically, during spectrum conversion, each frame of the spectrum corresponds one-to-one with a moment in the audio data. This correspondence allows the determination of the audio moment corresponding to each position in the spectrogram. Then, scale lines and values ​​are drawn at the corresponding positions according to the time scale of the audio data. For example, the scale value corresponding to the occurrence time is annotated at the energy pattern position corresponding to the acoustic event evidence content, ensuring a precise correspondence between the evidence content and the time scale within the spectrogram. By annotating the time scale position by position, the model can accurately determine the media moment corresponding to the acoustic evidence content based on the time scale in the spectrogram.

[0100] Another alternative method is to draw time scale bands corresponding to the audio data time scale in the edge region of the spectrogram, forming a spectrogram with a time axis. Specifically, time scale bands can be drawn at the lower or upper edge of the spectrogram, and scale lines corresponding to the audio data time scale can be arranged within the time scale bands, with corresponding scale values ​​labeled. The time scale bands and the energy pattern area of ​​the spectrogram belong to the same image object, the spectrogram. The time scale bands can be integrated with the spectrum... Figure 1 Similarly, it can be generated through rendering or drawn within the spectrogram after it has been generated. By drawing edge time scale bands, time coordinates can be presented without obscuring the energy pattern, allowing the model to simultaneously perceive the complete acoustic evidence content and the corresponding time scale.

[0101] In addition, in the segmented conversion scenario, the start and end scale values ​​of the corresponding audio segment in the audio data can be marked for each spectrogram, so that the time axis of each spectrogram is consistent with the time reference of the audio data, and a time-corresponding spectrogram sequence with time axis is obtained.

[0102] In one alternative embodiment, various visual presentation conditions can be designed for comparison and verification of the presentation format of the time coordinates.

[0103] Specifically, attached presentation conditions and contrastive presentation conditions can be constructed for the same audio data. Attached presentation conditions render the time scale of the audio data onto a spectrogram as described above, placing the time scale and the energy pattern carrying the acoustic evidence on the same spectrogram. Contrastive presentation conditions can be, for example, in the form of a digital clock, displaying a digital timestamp changing over time in the center of a completely black background; in one optional embodiment, the contrastive presentation condition is in the form of a progress bar, displaying a horizontal progress bar showing the fill ratio advancing over time; in another optional embodiment, the contrastive presentation condition is in the form of a moving light spot, displaying a light spot moving uniformly along a horizontal trajectory; in yet another optional embodiment, the contrastive presentation condition is in the form of a pure black video, displaying a completely black screen without time indication; in other optional embodiments, the contrastive presentation condition is in the form of direct audio input, where the audio data is directly used as model input without visualization.

[0104] The results show that presenting time coordinates and evidence content together in the same visual data can improve the accuracy of audio event time localization, while presenting time information and evidence content separately is not conducive to maintaining localization accuracy.

[0105] In the embodiments of this specification, the sound signal in the audio data is carried on a spectrogram through spectrum conversion, and the time scale of the audio data is rendered onto the spectrogram as a time coordinate to obtain a spectrogram with a time axis. This makes the time coordinate and the acoustic evidence content located in the same visual data, and the model can directly establish the correspondence between the acoustic evidence content and its time position. At the same time as identifying the audio event, the corresponding media moment can be determined, which improves the accuracy of audio event time positioning.

[0106] In one optional embodiment of this specification, the target multimodal data includes video data, and the visual data includes sampled frames; visual processing of the target multimodal data to obtain at least one visual data includes: Frame sampling is performed on the video data to obtain sampled frames; Rendering the time coordinates to visual data yields multimodal data with time coordinates, including: The timestamps of the video data are printed as time coordinates to the sampled frames to obtain sampled frames with timestamps.

[0107] The video branch is for locating visual events in the video, such as streetlights turning on or vehicles passing by. The video branch is rendered by printing timestamps on the sampled frames, so that the event and its time coordinates appear on the same image, without using a presentation method that overlays information streams independently.

[0108] Video data is multimodal data that carries continuously changing visual content over time. Specifically, video data is formed by the continuous playback of multiple frames in chronological order, and can take the form of surveillance recordings, screen recordings, video clips containing visual events, or real-time video streams. Video data may contain one or more visual events, which can serve as target objects to be located. Furthermore, there is a correspondence between the frames in the video data and the media timeline; the content changes as the media progresses, and the timeline corresponding to each frame can be determined based on the playback progress.

[0109] A sampled frame is an image frame obtained by sampling video data frame by frame. It is visual data formed after video data has been visualized. Specifically, sampled frames are obtained by extracting images corresponding to certain media moments from the video data according to a preset sampling frequency or preset sampling interval. The sampled frames are arranged in chronological order and aligned with the video data in time. The sampled frame carries the image content of the corresponding moment, that is, the evidence content of the target object in the video data, such as the image of a street lamp turning on. Furthermore, there can be one or more sampled frames, each of which corresponds one-to-one with a media moment in the video data. The sampled frame provides a printing medium for timestamps, and the timestamps are printed on the sampled frame and are located in the same frame as the evidence content.

[0110] A timestamp is a visible time marker representing a media moment, and it is the specific representation of time coordinates within a video branch. Specifically, a timestamp is used to represent the media moment corresponding to a sample frame, and can be presented in the form of visible characters, such as numerical text recording the moment. The media moment in the video data corresponding to a sample frame is the timestamp content printed onto that sample frame. Furthermore, printing the timestamp onto the sample frame means directly drawing it onto the image content of the sample frame, resulting in a timestamped sample frame. In a timestamped sample frame, the timestamp and the evidence content are located within the same image frame; the evidence content and its occurrence time can be read simultaneously on the same frame. A timestamped sample frame can serve as input content for multimodal data with time coordinates within a video branch.

[0111] In practical applications, frame sampling of video data to obtain sampled frames can be achieved through various methods.

[0112] One alternative method is to extract image frames from the video data as sampling frames at a preset sampling frequency. Specifically, sampling times can be determined on the timeline of the video data according to the preset sampling frequency, and the images corresponding to each sampling time can be extracted. The extracted image frames are arranged in chronological order and time-aligned with the video data. By uniformly extracting frames at the preset sampling frequency, the sampling frames can cover all time periods, reducing the possibility that the time when the evidence content is located was not sampled.

[0113] Another alternative method is to extract image frames as sample frames from the frame sequence of the video data at preset sampling intervals. Specifically, the video data can be decoded first to obtain a sequence of image frames arranged in playback order. Then, image frames can be extracted along this frame sequence at preset sampling intervals. For example, one frame can be extracted every preset number of image frames. Each sample frame maintains its sequential order in the frame sequence, forming a sample frame sequence consistent with the temporal order of the video data. By extracting frames at preset sampling intervals, temporally ordered sample frames can be directly obtained based on the frame arrangement, facilitating subsequent alignment with media moments in the video data.

[0114] Another alternative method is to extract image frames as sampling frames from keyframes in the video data. Specifically, video data typically contains keyframes during compression encoding. Keyframes can be decoded to obtain complete image content. Each keyframe in the video data can be located, decoded to obtain the corresponding image, and arranged in chronological order as sampling frames. When there are many keyframes, they can be further filtered using a preset sampling frequency or preset sampling interval. By extracting sampling frames from keyframes, the overhead of decoding and frame extraction processing can be reduced, improving the processing efficiency of obtaining sampling frames.

[0115] In addition, the time information corresponding to each sampled frame can be recorded at the same time as the frame sampling. For example, the media time or frame number corresponding to each sampled frame can be recorded so that the corresponding media time can be printed to each sampled frame later.

[0116] After obtaining the sampled frame containing the evidence of the target object, time-coordinate rendering can be further performed, printing the timestamp of the video data as a time coordinate onto the sampled frame, so that the evidence content and its occurrence time are in the same frame. In practical applications, printing the timestamp of the video data as a time coordinate onto the sampled frame to obtain a timestamped sampled frame can also be achieved through various methods.

[0117] One alternative approach is to directly print the timestamp of the video data as visible characters within a predetermined area of ​​the sampled frame. Specifically, a predetermined area for printing the timestamp can be determined on the sampled frame, such as the bottom or top of the screen. The timestamp corresponding to the media moment of the sampled frame is then drawn in this area, with the character content updated as the moment changes. The timestamp is directly drawn onto the content of the sampled frame, without generating a separate overlay image to carry the timestamp. By directly printing the timestamp onto the sampled frame, the timestamp and the evidence content can be located in the same frame, allowing the model to read the corresponding media moment while recognizing the evidence content.

[0118] Another alternative method is to print the timestamps of the video data onto the sampled frames according to a preset printing configuration. Specifically, the printing configuration can include the timestamp printing density, presentation format, and printing position. For example, the density can be set to one timestamp per unit of time or one timestamp every two units of time; the format can be in pure seconds or expanded by hours, minutes, and seconds; and the position can be selected as bottom center, bottom right, top center, top left, or top right, etc. By printing timestamps according to the preset printing configuration, the timestamp presentation method can be matched with application requirements, ensuring the readability of the timestamps while minimizing the impact on the content of the sampled frame.

[0119] Another alternative method is to determine the timestamp of the sampling frame based on the time information inherent in the video data itself, and then print it onto the sampling frame. Specifically, video data typically records the correspondence between frames and media timestamps. The time information corresponding to the sampling frame can be read according to the frame sequence number and used as the timestamp to print onto the sampling frame, ensuring that the timestamp matches the actual media timestamp in the video data. For video data that does not record this correspondence, the media timestamp corresponding to each sampling frame can be estimated based on the start time and playback progress and then printed onto the sampling frame. By determining the timestamp based on the time information inherent in the video data itself, an accurate temporal correspondence between the timestamp and the evidence content can be achieved, providing a reliable time reference for subsequent time positioning.

[0120] In addition, the printing method of the timestamp can be made to reduce the impact on the presentation of evidence content in the sample frame. For example, a smaller character size can be used, or the printing area can be set at the edge of the screen, so that the sample frame with the timestamp can retain the evidence content in the screen while presenting the timestamp.

[0121] In an alternative embodiment, the sampled frames can also be organized into a frame sequence in chronological order, and the frame sequence can be used as the content of multimodal data with time coordinates to input into the multimodal data processing model for inference in conjunction with the location request.

[0122] Specifically, the rendering process of the video branch has a positive impact on the model's temporal localization. There are rendering methods that can improve the accuracy of temporal localization on various models and various video localization evaluation data. The specific improvement effect is related to the combination of model and data. The appropriate rendering method can be selected through the conditional space screening described in the following embodiments. The rendering process does not involve updating the parameters of the multimodal data processing model.

[0123] In this embodiment, frame sampling is used to obtain sampled frames containing evidence of the target object from video data. The timestamp of the video data is directly printed onto the sampled frames as time coordinates, so that the timestamp and the evidence content are located in the same frame. The multimodal data processing model can read the corresponding media time while perceiving the evidence content, directly establishing the correspondence between the visual evidence content and its time position. This reduces the dependence on the time representation capability of the model during training, thereby improving the accuracy of video event time localization. Both frame sampling and timestamp printing are completed before model inference. The processing does not update the model parameters and can be applied to models that have already been trained or only provide an inference interface. This processing flow can support video event time localization in various deployment scenarios.

[0124] In an optional embodiment of this specification, before printing the timestamp of the video data as a time coordinate to the sampling frame, the method further includes: Based on the timestamp attribute parameters of the video data, a condition space for candidate rendering configurations is constructed. For each candidate rendering configuration in the condition space, the sampling frames with timestamps of video data printed according to each candidate rendering configuration are input into the multimodal data processing model along with the positioning request to perform inference and obtain the positioning evaluation result corresponding to each candidate rendering configuration. Based on the positioning evaluation results, the target rendering configuration is determined from the condition space; The timestamps of the video data are printed as time coordinates to the sampled frames to obtain timestamped sampled frames, including: According to the target rendering configuration, the timestamp of the video data is printed as a time coordinate to the sampling frame to obtain the sampling frame with timestamp.

[0125] For timestamp rendering of video branches, the optimal rendering method varies depending on the model and data combination. Before printing the timestamps of the video data as time coordinates to the sampled frames, a conditional space of candidate rendering configurations can be constructed based on the attribute parameters of the video data's timestamps. For each candidate rendering configuration in the conditional space, a multimodal data processing model is used for inference to obtain the localization evaluation results corresponding to each candidate rendering configuration. Based on this, the target rendering configuration is determined from the conditional space, and then printing is performed according to the target rendering configuration. By screening before printing, the timestamp rendering of the video branches can be matched with the actual model and data combination used.

[0126] Attribute parameters are configurable features that describe how timestamps are presented on sampling frames. Specifically, attribute parameters may include at least one of position, density, and format, where position defines the orientation of the timestamp drawn on the sampling frame, density defines the interval at which the timestamp appears in the sampling frame sequence, and format defines the presentation style of the timestamp content. Furthermore, the values ​​of the attribute parameters can be fixed after screening and determined, serving as a basis for repeated use under the same processing conditions in the future.

[0127] A candidate rendering configuration is a rendering configuration formed by a set of candidate values ​​for attribute parameters, used to indicate the specific presentation method when printing timestamps to sample frames. Specifically, one candidate value each for position, density, and format constitutes a candidate rendering configuration, corresponding to an executable rendering template. The differences between different candidate rendering configurations are reflected in the different positions of the timestamps in the sample frame image, the different density of the timestamps appearing in the sample frame sequence, or the different styles of timestamp content presentation. Furthermore, the candidate rendering configuration is the smallest enumeration unit in the condition space, used to determine the actual target rendering configuration after screening, while ensuring that the rendering form in the evaluation stage is consistent with the rendering form in the actual printing stage.

[0128] The condition space is a collection of candidate rendering configurations formed by combinations of candidate values ​​for various attribute parameters. Specifically, position candidate sets, density candidate sets, and format candidate sets can be used as combination dimensions. The candidate values ​​in each dimension are combined with each other, and the resulting candidate rendering configurations together constitute the condition space. The size of the condition space is determined by the number of candidate values ​​for each attribute parameter; the more candidate values, the more candidate rendering configurations the condition space contains. Furthermore, the condition space provides a complete enumeration range for rendering method screening. Its construction can be adapted to candidate values ​​by considering factors such as the image size of the sampled frame, the duration of the video data, and the readability of the timestamp, in order to control the size of the condition space. The condition space can also reuse the condition spaces constructed in the previous screening to avoid repeated construction.

[0129] The localization evaluation result is a metric characterizing the temporal localization performance of a multimodal data processing model when rendering timestamps according to a candidate rendering configuration. Specifically, the timestamps of sampled frames of the evaluation samples can be rendered according to the candidate rendering configuration. These timestamped sampled frames are then used to infer the localization request input to the multimodal data processing model. The time interval output by the model is compared with the real time interval corresponding to the target object on the evaluation sample. The degree of closeness between the compared time intervals constitutes the localization evaluation result for that candidate rendering configuration. For example, the intersection-union ratio (IUGR) can be used as a metric. The IUGR value is between zero and one; a higher value indicates a closer proximity between the localized interval and the real interval. Furthermore, the localization evaluation result is related to the set of evaluation samples and the model type used. The localization evaluation results for the same candidate rendering configuration may differ on different models or different data combinations. The localization evaluation process is completed offline before the formal temporal localization processing.

[0130] The target rendering configuration is the rendering configuration determined from the condition space used to actually print timestamps to the sampled frames. Specifically, the target rendering configuration can be a candidate rendering configuration whose positioning evaluation results meet preset selection criteria, such as the candidate rendering configuration with the highest value when using the intersection-union ratio (IU / R) metric. When the positioning evaluation results of multiple candidate rendering configurations are comparable, a selection can be made based on preset preference rules, such as prioritizing candidate rendering configurations with lower implementation costs or less impact on the content of the sampled frames. Furthermore, the target rendering configuration corresponds to the model and data combination used for screening. Printing timestamps according to the target rendering configuration ensures that the actual rendering form is consistent with the form verified in the evaluation phase, guaranteeing that the screening conclusions take effect in actual processing. The target rendering configuration can also be recorded along with the correspondence between the model and data combination for direct use when the same combination is reused.

[0131] In practical applications, constructing a condition space for candidate rendering configurations based on the timestamp attribute parameters of video data can be achieved through various methods.

[0132] One alternative approach is to combine the candidate values ​​of each attribute parameter, using the resulting set of candidate rendering configurations as the condition space. Specifically, the position candidate set, density candidate set, and format candidate set can be used as three combination dimensions. Each time, one candidate value is selected from each dimension for combination, and the resulting candidate rendering configurations collectively constitute the condition space. For example, if the position dimension contains multiple orientation candidates, the density dimension contains multiple interval candidates, and the format dimension contains multiple style candidates, all candidate rendering configurations can be obtained by combining them along these dimensions. By constructing the condition space through combination, the possible combinations of timestamp rendering methods can be covered more comprehensively, providing a full range of candidates for screening.

[0133] Another alternative approach is to adapt candidate values ​​to the characteristics of the sampled frames before combining them. Specifically, one can first obtain information such as the frame size, aspect ratio, and video duration of the sampled frames, then eliminate positional candidates that are incompatible with the frame size and density candidates that are incompatible with the video duration. The remaining candidate values ​​are then combined dimensionally to form a candidate rendering configuration, constituting a condition space; for example, positional candidates that cause timestamps to exceed the frame size can be eliminated. By adapting candidate values, the size of the condition space can be controlled, reducing meaningless candidate rendering configurations.

[0134] In addition, historically constructed condition spaces can be reused. For example, when the model and data combination is consistent with historical screening, the historical condition space can be directly used for evaluation to avoid repeated construction.

[0135] After constructing the condition space, it is necessary to evaluate the localization performance of each candidate rendering configuration in the condition space to obtain the localization evaluation result corresponding to each candidate rendering configuration. In practical applications, for each candidate rendering configuration in the condition space, the sampling frames with timestamps of video data printed according to each candidate rendering configuration are used to infer with the localization request input multimodal data processing model to obtain the localization evaluation result corresponding to each candidate rendering configuration. This can also be achieved through various methods.

[0136] One alternative approach is to sequentially perform rendering and inference evaluations on each candidate rendering configuration based on a pre-defined evaluation sample set. Specifically, the evaluation sample set includes multiple evaluation samples, each containing video data and the corresponding real-time interval of the target object. For each candidate rendering configuration, frame sampling is performed on the evaluation samples according to that configuration, and timestamps are printed on the sampled frames. The timestamped sampled frames are then input into a multimodal data processing model for inference along with the localization request. The output time interval is compared with the real-time interval to obtain the time interval proximity of the evaluation sample under that candidate rendering configuration. The time interval proximity of all evaluation samples in the evaluation sample set under that candidate rendering configuration is summarized, for example, by taking the average, to obtain the localization evaluation result of that candidate rendering configuration. By evaluating each configuration based on the evaluation sample set, the localization performance of each candidate rendering configuration can be compared using a unified sample caliber.

[0137] Another alternative approach is to share sampled frames to evaluate each candidate rendering configuration. Specifically, the video data used for evaluation can be sampled once to obtain a shared sampled frame for each evaluation sample. Then, for each candidate rendering configuration, timestamps are printed only on the shared sampled frame under that configuration, and the timestamps are input into a multimodal data processing model for inference to obtain the localization evaluation result for that configuration. Sharing sampled frames ensures the consistency of the image content and frame sequence during the evaluation of each candidate rendering configuration, with only the rendering form differing. By sharing sampled frames, the processing overhead of repeated frame extraction can be reduced, and the evaluation results of each candidate rendering configuration can be compared on the same image.

[0138] Furthermore, the evaluation process can be completed offline before the formal time-based localization process. For example, the evaluation of each candidate rendering configuration can be performed on a preset development sample, and the proximity of the time intervals on each sample can be summarized as the localization evaluation result of that configuration without occupying the inference resources of the formal processing. During the evaluation process, the prompt words, sampling and decoding parameters can be fixed so that the differences between each candidate rendering configuration come only from the rendering method itself.

[0139] After obtaining the localization evaluation results for each candidate rendering configuration, the target rendering configuration can be determined from the condition space. In practical applications, determining the target rendering configuration from the condition space based on each localization evaluation result can also be achieved through various methods.

[0140] One alternative approach is to compare the positioning evaluation results of each candidate rendering configuration and determine the candidate rendering configuration whose positioning evaluation results meet preset selection criteria as the target rendering configuration. Specifically, a unified metric can be used to compare the positioning evaluation results; for example, the candidate rendering configuration with the highest intersection-union ratio (IU) value can be determined as the target rendering configuration. When the positioning evaluation results of multiple candidate rendering configurations are comparable, preset preference rules can be used for selection, such as prioritizing the one with lower implementation costs. By comparing and selecting the best option, the rendering configuration can be easily filtered, resulting in a rendering method with better positioning performance.

[0141] Another alternative approach is to sort the location evaluation results of each candidate rendering configuration to obtain a ranking result, and then determine the target rendering configuration based on this ranking. Specifically, the candidate rendering configurations can be sorted from best to worst according to their location evaluation results, and the candidate rendering configuration ranked first can be determined as the target rendering configuration. Ties during the sorting process can be handled according to preset preference rules. Determining the target rendering configuration through ranking clearly presents the order of merit of each candidate rendering configuration, facilitating the review and adjustment of the screening results.

[0142] In addition, the correspondence between the target rendering configuration and the model and data combination can be recorded. When performing time positioning processing on the same model and data combination in the future, the recorded target rendering configuration can be used directly without repeated evaluation. When the model or data combination changes, the evaluation can be re-executed to determine the new target rendering configuration.

[0143] After determining the target rendering configuration, timestamp printing can be performed according to the target rendering configuration. In practical applications, the timestamps of the video data are printed as time coordinates to the sampling frames according to the target rendering configuration to obtain timestamped sampling frames. This can also be achieved through various methods.

[0144] One alternative approach is to perform printing according to the values ​​of the attribute parameters in the target rendering configuration. Specifically, the printing area for the timestamp can be determined based on the position value, the content format of the timestamp can be determined based on the format value, and the printing arrangement of the timestamp in the sampled frame sequence can be determined based on the density value. During printing, the timestamps corresponding to the media moments of each sampled frame are drawn onto the sampled frames according to the determined area, format, and arrangement, resulting in timestamped sampled frames. By printing according to the target rendering configuration, the screened and determined target rendering configuration can be verified, ensuring that the actual rendering form is consistent with the form verified in the evaluation phase.

[0145] Another alternative approach is to first obtain the configuration information corresponding to the target rendering configuration, and then perform printing according to the configuration information. Specifically, the target rendering configuration can be saved as configuration information containing the values ​​of each attribute parameter, such as saving the values ​​of position, density, and format in the form of structured records; before printing, the configuration information is read and parsed, and timestamped printing is performed according to the parsing results to obtain timestamped sample frames. By printing according to the configuration information, the target rendering configuration can be accurately used, and it is also convenient to reuse the configuration in the same processing flow.

[0146] Another alternative method is to dynamically adjust the printing method based on the content of the sampled frame during the printing process. Specifically, the printing area can be determined on the sampled frame first, and the positional relationship between the printing area and the area where the evidence content is located can be detected. If the two overlap, the printing area can be adjusted or the presentation size of the timestamp can be reduced to minimize the obstruction of the evidence content by the timestamp. After the adjustment is completed, printing can be performed. Through dynamic adjustment, the impact of the timestamp on the recognition of the evidence content can be reduced while adhering to the target rendering configuration.

[0147] In addition, the timestamped sampling frames can be verified after printing, such as checking whether the timestamps are clear and readable, whether they obscure the evidence content, and making local fine adjustments if necessary, to ensure the quality of the timestamped sampling frames entering the multimodal data processing model.

[0148] In an optional embodiment, the accumulated comparison results of timestamp rendering can also be referenced to provide a reference for the selection of rendering configurations for subsequent combinations of the same model and data. For example, dense timestamps are generally better than sparse timestamps, pure second format is generally better than format expanded by hour, minute, and second, and central position is generally better than side position. However, the target rendering configuration changes with the model and data combination, and there is no universal rendering template. Therefore, the target rendering configuration should be determined according to the actual model and data combination used through conditional space screening so that the rendering method matches the actual processing conditions.

[0149] In the embodiments of this specification, before printing the timestamps of video data to the sampling frames, a conditional space for candidate rendering configurations is constructed based on the attribute parameters of the timestamps. A multimodal data processing model is used to infer the localization evaluation results for each candidate rendering configuration in the conditional space. Based on these localization evaluation results, the target rendering configuration is determined from the conditional space, and printing is then performed according to the target rendering configuration. This allows the timestamp rendering method of the video branch to be combined with model and data combinations for screening and adaptation, rather than using a fixed rendering method for different model and data combinations. This improves the localization gain brought by timestamp rendering and enhances the robustness of time localization processing under different model and data combinations. Simultaneously, by completing the evaluation offline and determining the target rendering configuration through comparison, sorting, or initial screening, and recording the target rendering configuration corresponding to the model and data combination, the processing overhead of repeated evaluations is reduced, improving the efficiency of rendering configuration determination in multi-model, multi-data combination scenarios.

[0150] In an optional embodiment of this specification, before inputting the time-coordinated multimodal data and the positioning request into a preset multimodal data processing model, the method further includes: Freeze the parameters of the multimodal data processing model; Inference is performed on multimodal data with time coordinates based on the location request to obtain the time interval corresponding to the target object in the target multimodal data, including: Using a multimodal data processing model with frozen parameters, forward inference is performed on multimodal data with time coordinates based on the location request to obtain the time interval corresponding to the target object in the target multimodal data. During the forward inference process, the parameters of the multimodal data processing model remain unchanged.

[0151] Before inputting time-coordinated multimodal data and location requests into the multimodal data processing model, the model's parameters can be frozen. In the frozen state, the model's weights retain their values ​​from when loading was complete and do not participate in any updates during subsequent inference. The model only performs forward inference from input to output and does not perform training, fine-tuning, or other parameter adjustments. The freezing process only affects the model's operating mode and does not change the model's weights or structure; it is unrelated to the model's training process.

[0152] In practical applications, freezing the parameters of a multimodal data processing model can be achieved through various methods.

[0153] One alternative approach is to load the weights of a pre-trained multimodal data processing model and configure the model to run in inference mode, placing the model weights in a read-only state. Specifically, in inference mode, the model only retains the forward computation path from input to output; the weights reside read-only in the runtime environment, intermediate computation results do not flow back to the weights, and there is no step to update the weights. For example, the model can be configured to always remain in inference mode, without executing training-related processes. By running the model in inference mode, the weights can remain unchanged during subsequent inference, achieving parameter freezing.

[0154] Another alternative approach is to load model weights in read-only mode through an inference service, providing forward inference capabilities to external systems. Specifically, the trained multimodal data processing model and its weights can be loaded into the inference service in read-only mode. The inference service receives multimodal data with time coordinates and location requests, invokes the model to perform forward inference, and returns the inference output. During the inference service's operation, the weights are not writable; the caller receives the forward inference output. By using an inference service to handle forward inference, the weights can be restricted to read-only mode at the service runtime level, keeping the model parameters frozen during continuous service operation.

[0155] Another alternative approach is to permanently save the multimodal data processing model after training and consistently use this saved model for inference in subsequent processing. Specifically, after training, the model and its weights can be exported as a deployment file that retains only forward inference capabilities, excluding the update configurations from the training phase. When inference is needed, the model is loaded from the deployment file, providing a model with its parameters frozen. By permanently saving the trained model and loading based on the saved result, the parameter update mechanism can be avoided by entering the running state along with the model, thus achieving stable parameter freezing.

[0156] In addition, the parameter freezing status can be confirmed before inference. For example, the weights of the current model can be verified to be consistent with the fixed weights before forward inference can begin.

[0157] After freezing the model parameters, inference can be performed based on the location request. In practical applications, using a multimodal data processing model with frozen parameters, forward inference can be performed on multimodal data with time coordinates based on the location request to obtain the time interval corresponding to the target object in the target multimodal data. This can also be achieved through various methods.

[0158] One alternative approach is to perform a forward inference on the multimodal data with time coordinates and the location request, determining the time interval corresponding to the target object from the model output. Specifically, the implementation of identifying the evidence content of the target object and reading the visible time coordinates during the inference process, and determining the time interval from the model output, is consistent with the corresponding steps in the aforementioned embodiments. The difference is that in this embodiment, the inference is performed based on a multimodal data processing model with frozen parameters. The model parameters remain unchanged throughout the inference process, and the output is entirely determined by the input data and the frozen parameters. Obtaining the time interval directly through a single forward inference allows time location processing to be completed with a concise processing path.

[0159] Another alternative approach is to perform multiple forward inferences on the input and obtain the time interval corresponding to the target object from the outputs of these multiple inferences. Specifically, when multimodal data with time coordinates contains multiple target objects, the input can be organized by target object, and forward inference can be performed separately to obtain the time interval corresponding to each target object from each output. When the time range corresponding to the target object is long, the input can also be divided into time windows and forward inference can be performed in stages. The results of each stage can be aligned by time coordinates and combined to obtain the complete time interval. Each forward inference is performed based on the same set of frozen parameters, and no parameter updates are generated during the inference process. The same model can process different target multimodal data sequentially. By combining multiple forward inferences, localization scenarios containing multiple target objects or long time ranges can be supported.

[0160] Another alternative approach is to delegate forward inference to a high-throughput inference engine and implement read-only intervention for processing stages requiring runtime intervention. Specifically, for inference paths that do not require additional intervention, forward inference can be performed by the high-throughput inference engine to improve inference speed. For processing stages requiring intervention, the hook mechanism of the deep learning framework can be used to read or adjust intermediate data of the corresponding stage during inference. This intervention applies to the data during the forward computation process without modifying model weights or changing the model structure; the model parameters remain frozen. Through the combination of the high-throughput inference engine and the hook mechanism, forward inference can be completed while parameters are frozen, balancing processing adaptability and inference efficiency.

[0161] Furthermore, the multimodal data processing model with frozen parameters retains its original parameters after completing one forward inference, and can be directly used for other localization processes. For example, in batch processing scenarios, different target multimodal data and corresponding localization requests can be sent to the same model to perform forward inference one after another, without the need to reload or adjust the model between each inference.

[0162] In an alternative embodiment, the above processing can also be implemented as capability enhancement middleware, mounted on top of an already deployed frozen model.

[0163] In one optional embodiment, the frozen model is a self-developed open-source model; in another optional embodiment, it is a third-party model that only provides an inference interface. As long as the input and output interfaces of the frozen model meet the processing requirements, multimodal data with time coordinates and positioning requests can be input to it, and outputs used to determine the time interval can be obtained. The entire processing does not require the model provider to cooperate in training or fine-tuning, and the mounting process does not change the parameters and structure of the frozen model itself. Furthermore, both visualization processing and time coordinate rendering are deterministic processes, without introducing learnable parameters. The parameter state of the entire processing chain remains consistent, enabling smooth integration with the frozen model.

[0164] In the embodiments described in this specification, the parameters of the multimodal data processing model are frozen before the multimodal data with time coordinates and the positioning request are input into the multimodal data processing model. The multimodal data processing model with frozen parameters is then used to perform forward inference on the multimodal data with time coordinates based on the positioning request while keeping the parameters unchanged, thereby obtaining the time interval corresponding to the target object in the target multimodal data. This allows the time positioning processing to be completed without updating any model parameters. Furthermore, by explicitly freezing the model parameters, the data preparation and computational resource overhead required for training or fine-tuning are avoided. It also facilitates the integration of the processing process into the deployed model service as middleware, thereby improving the feasibility of time positioning processing in a practical system.

[0165] In one optional embodiment of this specification, inference is performed on multimodal data with time coordinates based on a location request to obtain the time interval corresponding to the target object in the target multimodal data, including: Inference is performed on multimodal data with time coordinates based on the location request to obtain the initial time interval of the target object; Based on the initial time interval, determine the time interval corresponding to the target object in the target multimodal data.

[0166] The initial time interval is the first-hand time localization result obtained by inferring from multimodal data with time coordinates based on the localization request. It can be understood as the raw localization result without repair or further filtering. Specifically, the initial time interval is directly derived from the inference performed on the multimodal data with time coordinates and is used to characterize the time range in which the target object appears or occurs in the target multimodal data. In the process of obtaining the initial time interval, no additional processing such as repair or further filtering is performed on the inference output. The initial time interval can reflect the original localization judgment of the target object's time position by the multimodal data processing model.

[0167] Furthermore, the initial time interval provides a baseline result for possible subsequent processing. The time interval corresponding to the target object in the target multimodal data determined based on the initial time interval can either be directly used or obtained by further processing based on the initial time interval. The difference between the initial time interval and the finally determined time interval reserves processing space for further processing as needed.

[0168] In practical applications, the initial time interval of the target object can be obtained by reasoning from multimodal data with time coordinates based on the location request through various methods.

[0169] One alternative approach is to perform direct inference on multimodal data with time coordinates based on the location request, using the first-hand time positioning result obtained through inference as the initial time interval for the target object. Specifically, inference can be performed as described in step 108, using the multimodal data with time coordinates and the location request as input to the multimodal data processing model. The model determines the time position corresponding to the target object based on the target object indicated by the location request. After inference, no repair or further filtering is performed on the output result; the original time positioning result obtained through inference is directly retained as the initial time interval. By performing direct inference and retaining the inference result, the initial time interval can fully reflect the model's original location judgment of the target object's time position, providing a usable baseline result for subsequently determining the time interval corresponding to the target object.

[0170] Another alternative approach is to organize the inference process for obtaining the initial time interval as an independent intermediate step. The inference output is first saved as the initial time interval before proceeding to the final time interval determination stage. Specifically, inference can be performed as described in the preceding embodiments. After inference, the obtained initial time interval is temporarily stored as an intermediate result. This temporarily stored initial time interval is not directly used as the final output. It can be used later to directly determine the time interval corresponding to the target object, or further verification or comparison can be triggered on the initial time interval. The initial time interval obtained from a single inference can be reused for different determination methods. By organizing the inference process as an independent intermediate step, an entry point can be reserved for subsequent, on-demand processing. This allows for direct determination using the initial time interval when no further processing is needed, maintaining a concise processing flow.

[0171] In addition, other equivalent inference organization methods can be used to obtain the initial time interval, as long as the inference can process multimodal data with time coordinates based on the location request and retain the first-hand time location results that have not been repaired or further filtered.

[0172] After obtaining the initial time interval of the target object, the corresponding time interval of the target object in the target multimodal data can be determined based on the initial time interval. This determination step is used to obtain the final positioning result output by the time positioning processing. In practical applications, determining the corresponding time interval of the target object in the target multimodal data based on the initial time interval can also be achieved through various methods.

[0173] One alternative approach is to directly define the initial time interval as the time interval corresponding to the target object in the target multimodal data. Specifically, if the initial time interval is reliable and requires no further processing, it can be directly output as the time interval corresponding to the target object. The final determined time interval is the same as the initial time interval, and the determination process does not include any additional processing steps beyond inference. By directly using the initial time interval, unnecessary processing overhead can be omitted, keeping the time localization process simple and efficient, making it suitable for fast localization scenarios with high processing speed requirements.

[0174] Another alternative approach is to first verify or validate the initial time interval, and then determine the corresponding time interval for the target object in the target multimodal data based on the verification or validation results. Specifically, a verification process can be performed on the initial time interval to confirm its suitability as the time interval corresponding to the target object. The specific implementation of the verification rules can be determined according to actual application requirements and will not be elaborated here. If the initial time interval passes verification, it is determined as the time interval corresponding to the target object; if the initial time interval fails verification, further processing of the initial time interval is triggered to obtain a usable time interval. By verifying first and then determining, a flexible choice can be made between direct adoption and further processing, ensuring that the output positioning results undergo necessary checks, balancing processing efficiency and output reliability.

[0175] Another alternative method is to combine the anchor point prediction results of the target object's time position to determine the corresponding time interval of the target object in the target multimodal data. The anchor point prediction results can be the prediction results obtained by inference on the target multimodal data without superimposed time coordinates, as described in the embodiments below. Specifically, the anchor point prediction results can be compared with the initial time interval. If the anchor point prediction results match the initial time interval, the initial time interval is directly used as the time interval corresponding to the target object; if they do not match, further processing is triggered. For example, the initial time interval can be reviewed, or the time interval corresponding to the target object can be further processed according to the method described in the embodiments below based on the difference between the anchor point prediction results and the initial time interval. By combining the anchor point prediction results for comparison, the initial time interval can be cross-verified while maintaining processing simplicity, improving the accuracy of the determined time interval.

[0176] In addition, the above methods can be combined. For example, the initial time interval can be verified first. If the verification passes, the initial time interval can be used directly. If the verification fails, the anchor point prediction results can be compared and corresponding processing can be triggered to adapt to the positioning processing needs of different scenarios.

[0177] In an alternative embodiment, after obtaining the initial time interval through inference, it may be decided whether to enable further processing based on the deployment scenario.

[0178] For example, in deployment scenarios where only rapid location is required, the initial time interval can be directly used as the time interval corresponding to the target object in the target multimodal data. In deployment scenarios where positioning accuracy is sensitive, further processing can be enabled, using methods such as verification before adoption or comparison with anchor point prediction results to determine the time interval corresponding to the target object based on the initial time interval. Furthermore, the initial time interval can also serve as tag-free reference information for subsequent processing.

[0179] In the embodiments of this specification, the initial time interval of the target object is obtained by inferring from multimodal data with time coordinates based on the positioning request. Then, the time interval corresponding to the target object in the target multimodal data is determined based on the initial time interval. The processing of the interval obtained by inference is decoupled from the processing of determining the final interval. The initial time interval serves as an intermediate result connecting the two, providing an entry point for further processing as needed. Accordingly, the final determined time interval can be the same as the initial time interval or it can be the result of further processing based on the initial time interval. When further processing is not enabled, the processing flow remains simple and efficient. When further processing is enabled, the processing flow can be flexibly adjusted according to the application scenario, so that the time positioning processing balances processing efficiency and processing flexibility.

[0180] In an optional embodiment of this specification, before determining the time interval corresponding to the target object in the target multimodal data based on the initial time interval, the method further includes: The initial time interval is validated based on preset validation rules. Based on the initial time interval, determine the corresponding time interval of the target object in the target multimodal data, including: If the legality verification passes, the time interval corresponding to the target object in the target multimodal data is determined based on the initial time interval.

[0181] Preset validation rules are pre-defined rules used to determine whether the initial time interval is valid. Specifically, the boundary information of the initial time interval can be the start time and the end time. The preset validation rules can set validation content for the boundary information, which may include: the end time is not earlier than the start time, neither the start time nor the end time exceeds the duration range corresponding to the target multimodal data, and it can be successfully parsed according to the agreed format, etc. Furthermore, the preset validation rules can be one or more of the above validation content, and the combination method and boundary conditions can be pre-configured, such as whether the boundary time is allowed to be equal to the duration boundary of the target multimodal data.

[0182] Legality verification is a validation process based on preset verification rules to check whether the initial time interval meets the legality requirements. Specifically, legality verification uses the boundary information of the initial time interval as the verification object. It can be checked item by item according to the preset verification rules. If all verification items are satisfied, the verification is considered passed; otherwise, the verification is considered failed. When the verification is passed, the initial time interval can be used as the basis for determining the corresponding time interval of the target object in the target multimodal data. Furthermore, the initial time interval can be obtained from the text inference results output by the multimodal data processing model. Legality verification can be performed after extracting boundary information from the text inference results according to the agreed format. Legality verification is a deterministic post-processing of output and does not involve learnable parameters.

[0183] In practical applications, the validity of the initial time interval can be verified using a variety of methods based on preset verification rules.

[0184] One alternative method is to verify, based on preset validation rules, that the end time of the initial time interval is not earlier than the start time. Specifically, the start and end times of the initial time interval can be read, and their order on the timeline can be compared. If the end time is not earlier than the start time, the validation entry is deemed valid; otherwise, it is deemed invalid. By verifying the order of boundary times, inverted initial time intervals can be identified. This allows for time sequence checks before determining the time interval corresponding to the target object, thus validating the time sequence validity of the initial time interval.

[0185] Another alternative method is to verify, based on preset validation rules, that the start and end times do not exceed the duration range of the target multimodal data. Specifically, the duration range corresponding to the target multimodal data can be obtained, and the start and end times can be compared to this range. If both do not exceed the duration range, the validation entry passes; otherwise, it fails. Whether boundary times are allowed to equal the endpoint values ​​of the duration range can be pre-configured. For example, if the boundary is allowed to equal the duration boundary, an initial time interval where the end time is the same as the end time of the target multimodal data can still be considered passable. By verifying whether boundary times exceed the limits, the initial time interval can be constrained to the actual time range covered by the target multimodal data, blocking out-of-bounds intervals.

[0186] Another alternative method is to verify whether the initial time interval can be successfully parsed according to a pre-defined verification rule. Specifically, the text inference result is the natural language text output by the multimodal data processing model. The start and end times of the target object can be extracted from the text inference result according to a pre-defined format. If the complete boundary information is successfully parsed and obtained, the verification content is deemed to have passed; if the start and end times cannot be parsed, the verification content is deemed to have failed. By verifying whether the parsing according to the pre-defined format is successful, the initial time interval from which complete boundary information cannot be recovered can be identified, thus ensuring the legitimacy of the source.

[0187] In addition, the above-mentioned multiple verification contents can be combined to form a preset verification rule. For example, first verify whether it can be parsed successfully according to the agreed format, then verify the timing relationship and duration range of the boundary moments, and only when all verification contents pass can the legality verification be determined to be passed.

[0188] If the legitimacy check passes, the initial time interval has a valid and usable interval format, and the process can proceed to the step of determining the corresponding time interval of the target object based on the initial time interval. This step is used to obtain the output result of this positioning. In practical applications, determining the corresponding time interval of the target object in the target multimodal data based on the initial time interval, after the legitimacy check passes, can also be achieved through various methods.

[0189] One alternative approach is to directly determine the initial time interval as the time interval corresponding to the target object in the target multimodal data, provided that the validity check passes. Specifically, a successful validity check indicates that the initial time interval meets the preset validation rules, and this initial time interval can be directly used as the time interval corresponding to the target object in the target multimodal data, serving as the output result of this localization. By directly using the validated initial time interval, invalid initial time intervals can be avoided from being output as localization results, while maintaining a simple determination process and determining the time interval corresponding to the target object with minimal processing overhead.

[0190] Another alternative approach is to process the initial time interval according to the expression requirements of the target object's corresponding time interval to determine the target object's time interval. Specifically, the expression format of the time interval can be predetermined according to application needs. For example, if a relative position interval within the media duration is to be used, the boundary times of the initial time interval can be converted into relative positions within the media duration, and the processed interval can be determined as the time interval corresponding to the target object. If the intended expression format is consistent with the initial time interval, the initial time interval can be used directly. By processing the initial time interval according to the expression requirements, it can be ensured that the determined time interval expression is standardized and consistent with the time system of the target multimodal data, facilitating direct use in subsequent processing.

[0191] In addition, the above methods can be used in combination. For example, it can be determined whether the expression of the initial time interval is consistent with the expression requirements of the intended output time interval. If they are consistent, they can be used directly; if they are inconsistent, they can be converted and then used.

[0192] In an alternative embodiment, the case where the validity check fails can also be handled.

[0193] Specifically, when the legality check fails, the processing of determining the time interval corresponding to the target object can be omitted from the initial time interval, and the corresponding text reasoning result can be marked as an unlocatable result or invalid output. For example, it can be recorded as an invalid result and counted separately when evaluating the positioning effect. Alternatively, re-reasoning can be triggered according to a preset strategy. For example, reasoning can be performed again on the same multimodal data with time coordinates and positioning request, or subsequent backup processing can be enabled. The specific implementation methods of re-reasoning and backup processing can be found in the description of the relevant embodiments below.

[0194] In this embodiment, before determining the time interval corresponding to the target object in the target multimodal data based on the initial time interval, the validity of the initial time interval is verified based on preset verification rules. If the verification passes, the time interval corresponding to the target object is then determined based on the initial time interval. This ensures that the initial time interval undergoes a validity check before entering the determination stage. Illegal initial time intervals will not flow into the determination stage and will not be output as positioning results, guaranteeing that the output time interval format is valid and can be directly used for subsequent processing. At the same time, the determination process is only executed when the initial time interval has a valid format, providing a standardized interval format for the positioning output, improving the usability of the positioning output, and also providing a standardized basis for subsequent processing.

[0195] In one optional embodiment of this specification, inference is performed on multimodal data with time coordinates based on a location request to obtain the initial time interval of the target object, including: Based on the location request, inference is performed on multimodal data with time coordinates to obtain text inference results; the text inference results are parsed to obtain the initial time interval of the target object.

[0196] The text inference result is the result obtained by inferring the time and location information of the target object in the form of natural language text after considering the multimodal data with time coordinates based on the location request. Specifically, the text inference result is the text response generated by the multimodal data processing model in response to the location request. The response carries a description of the start and end times of the target object, such as "started at... ended at..." or "located at... until... seconds". The model generates the response text word by word in an autoregressive manner, and the output content is the model's natural language response to the location request.

[0197] Furthermore, the text reasoning result is an intermediate reasoning product between reasoning and parsing. The initial time interval is obtained by parsing the text reasoning result. The multimodal data processing model on which the reasoning is based is a trained model with unchanged parameters. Its output is text, and there is no output structure that can directly regress the numerical interval. The text reasoning result is the text carrier of the model carrying the time positioning judgment. The wording of the time description can be agreed upon in advance or expressed freely.

[0198] In practical applications, reasoning about multimodal data with time coordinates based on location requests to obtain text reasoning results can be achieved through various methods.

[0199] One alternative approach is to input multimodal data with time coordinates and location requests into a multimodal data processing model, allowing the model to directly generate a natural language response containing the target object's time and location information. Specifically, the model encodes the rendered visual data with time coordinates, combines this with evidence from the location request indicating the target object, reads the corresponding time coordinates, and organizes the determined start and end times into natural language, generating a word-by-word output. The resulting response constitutes the text-based reasoning result. By directly generating natural language responses from the model, reasoning can be completed using the model's existing text generation capabilities.

[0200] Another alternative approach is to pre-define the format of time descriptions in the model's response during inference, allowing the model to generate responses according to this pre-defined format, and then using these responses as the text inference result. Specifically, a time description convention can be appended to the input content fed into the model. For example, the model can be required to provide the start and end times of the target object in its response using a pre-defined sentence structure. When generating the response, the model organizes the text according to this convention, resulting in a text inference result with well-defined time descriptions. By pre-defining the format of time descriptions, the position of time descriptions in the text inference result can be relatively fixed, facilitating parsing and processing.

[0201] Another alternative approach is to leave the expression of time descriptions in the model's response unrestricted, allowing the model to freely organize its responses and use these freely generated responses as the text inference result. Specifically, the model autonomously determines how to describe time based on the semantics of the location request; for example, it can use different wording such as "from...to...end" to describe the start and end times. As long as the response contains the time and location information of the target object, it can be used as the text inference result. By allowing the model to freely express time descriptions, no additional constraints need to be imposed on the model's output.

[0202] Furthermore, when the multimodal data with time coordinates is long, inference can be performed on the multimodal data with time coordinates in segments according to time windows, and the answers obtained from each segment of inference can be combined in chronological order to obtain text inference results that cover the entire time range.

[0203] After obtaining the text inference results, since the model output carries the start and end time description of the target object in text, the model is required to directly output a structured time interval. This requires changing the output structure of the model or performing additional training, which is difficult to apply to models with unchanged parameters or that only provide an inference interface. Accordingly, the text inference results can be parsed to convert the start and end time description carried in the text into a structured time range, that is, to obtain the initial time interval of the target object.

[0204] In practical applications, obtaining the initial time interval of the target object by parsing the text reasoning results can also be achieved through various methods.

[0205] One alternative approach is to extract the start and end times and corresponding time units from the text inference results based on a preset template or rules to obtain the initial time interval of the target object. Specifically, the position of the time description in the text inference results can be located according to the preset template, and the start and end times and their corresponding time units can be read from the corresponding positions, converted into start and end times under a unified measurement method, and thus constitute the initial time interval; the parsing process does not include learnable processing parameters. By extracting time numbers and units based on preset templates or rules, the initial time interval can be stably obtained from the well-organized text inference results.

[0206] Another alternative approach is to identify start and end time descriptions from freely expressed textual reasoning results based on time phrase recognition, thereby obtaining the initial time interval of the target object. Specifically, the textual reasoning results can be segmented and semantically analyzed to identify time phrases representing start or end times. Time values ​​can be extracted and combined with phrase semantics to determine the corresponding times, thus establishing the start and end times as the initial time interval. By performing parsing based on time phrase recognition, it is possible to accommodate various free expressions of time in the model, expanding the applicability of the parsing process.

[0207] Another alternative method is to perform a parsing check on the text reasoning results. Unparseable text reasoning results are treated as invalid, and the reasoning is re-executed before parsing again to obtain the initial time interval of the target object. Specifically, during parsing, it can first be determined whether the text reasoning result contains a identifiable time description. If it does, the start and end times are extracted according to parsing rules to form the initial time interval. If it does not contain the description or the extraction result is incomplete, the text reasoning result can be treated as invalid output, or the reasoning can be re-executed, and the regenerated text reasoning result can be parsed again. By combining invalidation processing with re-reasoning, the usability of the parsing process can be ensured, and the initial time interval can be further validated after parsing.

[0208] In addition, parsing based on preset templates or rules can be combined with parsing based on time phrase recognition. For example, the text reasoning results can be matched with preset templates first, and then parsing can be performed by time phrase recognition when the template is not matched, thus balancing parsing speed and applicability.

[0209] In an alternative embodiment, the parsing format convention can also be carried in the input to improve the stability of parsing.

[0210] Specifically, output format requirements can be included in the location request, requiring the model to provide the start and end times of the target object in the response according to a pre-defined separator format. For example, the model can be required to place the start and end times in a pre-defined position in the response and distinguish them from other content using a pre-defined separator. During parsing, locating and extracting the time field according to the same convention can reduce the impact of wording variations on parsing. Furthermore, there is a parsing correspondence between the text inference result and the initial time interval, with the parsed initial time interval consistent with the time description in the text inference result.

[0211] In the embodiments of this specification, inference is performed on multimodal data with time coordinates based on the location request to obtain text inference results carrying the time location information of the target object. Then, the text inference results are parsed to obtain the initial time interval of the target object. With text as the carrier of model output, the time location processing is compatible with autoregressive generative models with text as the output, without modifying the model's output structure or performing additional training on the model. It can be applied to models with unchanged parameters or only providing an inference interface. The parsing process is a deterministic post-processing, which provides a structured initial result for subsequently determining the time interval corresponding to the target object in the target multimodal data.

[0212] In an optional embodiment of this specification, after inferring the initial time interval of the target object from the multimodal data with time coordinates based on the location request, the method further includes: Obtain the hidden states of each word generated during the inference process of the multimodal data processing model; Determine the readability of the target object's temporal information in the hidden state, and the coordinate compatibility of the temporal information among the various modalities of the target's multimodal data; The target intervention layer is determined based on readability and coordinate compatibility. Based on the initial time interval, determine the corresponding time interval of the target object in the target multimodal data, including: Based on the target intervention layer and the initial time interval, the time interval corresponding to the target object in the target multimodal data is determined.

[0213] After inferring the target object from the multimodal data with time coordinates based on the location request and obtaining the initial time interval of the target object, the hidden state in the inference process can be used for diagnosis to determine the target intervention layer. Based on the target intervention layer and the initial time interval, the corresponding time interval of the target object in the target multimodal data can be determined.

[0214] A lexical is a basic unit segmented when the model processes input. Both visual and audio inputs enter the multimodal data processing model in the form of lexical units. Specifically, local regions in visual data can correspond to visual lexical units, and content segments in audio data can correspond to audio lexical units. Furthermore, lexical units are correlated with their temporal positions in the media.

[0215] The hidden state is the internal representation generated by each layer for each word during the forward inference process of a multimodal data processing model. Specifically, each layer generates a corresponding hidden state for each word, and the hidden states are organized according to the model layer and the word position. Furthermore, the hidden state can be read during the inference process through the model's inference interface, and the reading process does not affect the model parameters or normal inference.

[0216] Temporal information refers to information related to the temporal location of a target object within the media. Specifically, the time range in which the target object appears or occurs will form corresponding encoded content within the model. For example, the encoded information corresponding to a specific moment in the media is temporal information. This type of information can be distributed in the latent states of words related to the target object or words corresponding to time coordinates.

[0217] Readability is a metric used to measure whether the temporal information of a target object can be read from its latent state. Specifically, readability is high when temporal information is encoded in the latent state in a relatively direct form; and low when it requires complex transformations to extract. Furthermore, readability is determined layer by layer, and the results of each layer constitute a readability distribution, which supports the determination of the target intervention layer.

[0218] Coordinate compatibility is a measure of how well temporal information can be shared and used across different modalities of a target multimodal data set. Specifically, coordinate compatibility is high when the readout method obtained from the latent state of one modality can be applied unchanged to the latent state of another modality; otherwise, it is low. Furthermore, coordinate compatibility is independent of readability; temporal information being readable in a single modality does not guarantee its usability when shared across different modalities.

[0219] The target intervention layer is a model layer determined within a multimodal data processing model for implementing intervention processing. Specifically, intervention processing involves additional processing of data at a specific layer during the inference process, without altering the model's parameters or structure. Furthermore, the target intervention layer is determined based on readability and coordinate compatibility.

[0220] In practical applications, the hidden states of each word generated during the inference process of a multimodal data processing model can be obtained through various methods.

[0221] One alternative approach is to use the model's hook mechanism to read the hidden state of a specified layer during inference. Specifically, hooks are registered on the layers to be read. When inference passes through that layer, the hooks are triggered, and the hidden state output by the layer is transferred to a pre-defined storage area. When reading multiple layers, hooks can be registered on each layer separately and then removed after inference is complete. This hook mechanism allows the hidden state to be obtained without modifying the model structure and weights.

[0222] Another alternative approach is to obtain the hidden state through an inference interface that supports intermediate layer output. Specifically, when invoking inference, the layer information for which the hidden state needs to be returned can be carried in the interface parameters, and the hidden state of the corresponding layer can be parsed from the interface return result; when deployed as a service, the inference service can also output the hidden state of the specified layer.

[0223] In addition, a separate inference can be performed for diagnosis, and the hidden state of each word can be obtained during the inference process. When obtaining the state, the correspondence between the word and the time position should be maintained. For example, the modality to which each word belongs and the corresponding media time should be recorded and stored in an organized manner according to the layer and word position.

[0224] After obtaining the hidden state of each lexical unit, the readability of the target object's time information in the hidden state can be further determined. In practical applications, determining the readability of the target object's time information in the hidden state can also be achieved through various methods.

[0225] One alternative approach is to train probes for reading out time information on the hidden states, and determine the readability based on the probes' reading performance. Specifically, using the hidden states of each layer as input and the normalized values ​​of the corresponding media moments as the supervision target, probes for reading out time values ​​from the hidden states are trained on each candidate layer, and the prediction performance of each layer's probes is evaluated using metrics such as the coefficient of determination. A higher coefficient of determination indicates a higher readability; thus, the distribution of readability in each layer's hidden states can be obtained.

[0226] Another alternative approach is to determine readability by analyzing the distribution characteristics of latent states as they change with media time. Specifically, when latent states carry readable temporal information, the latent states of words adjacent to the media time have smaller differences in the temporal correlation direction, while the latent states of words farther apart have larger differences in that direction. The consistency between the differences in latent states and the media time interval is statistically analyzed; stronger consistency indicates higher readability.

[0227] Furthermore, readability can be determined separately for different modalities of lexical units, and the results are organized into a readability distribution layer for use when determining the target intervention layer.

[0228] After determining readability, coordinate compatibility also needs to be determined. In practical applications, determining the coordinate compatibility of time information across different modes of the target multimodal data can be achieved through various methods.

[0229] One alternative approach is to determine coordinate compatibility using a cross-modal transfer test. Specifically, a probe for reading out time information is trained based on the latent states of the units corresponding to the time coordinates in the visual data. This probe is then applied to the latent states of the units corresponding to the audio data and compared with the baseline prediction results. The baseline prediction results output time values ​​based solely on the modality's own statistical information, such as the mean of the modality set. If the probe performs worse than the baseline prediction results, a negative cross-modal transfer occurs, indicating a lack of shared and usable coordinate representations between modalities and low coordinate compatibility; conversely, a higher degree of compatibility indicates greater compatibility.

[0230] Another alternative approach is to compare the readouts from different modalities at the same time point. Specifically, cross-modal words corresponding to media moments are selected, and time values ​​are read from the latent states of each modality. These values ​​are then converted to a unified time coordinate system for comparison. When the values ​​are close to each other, the coordinate compatibility is high; when there are systematic differences, the coordinate compatibility is low.

[0231] In addition, cross-modal tests can be performed at multiple media moments and on multiple layers, and the coordinate compatibility can be determined by combining the results of each test.

[0232] After determining readability and coordinate compatibility, the target intervention layer can be determined accordingly. In practical applications, determining the target intervention layer based on readability and coordinate compatibility can also be achieved through various methods.

[0233] One alternative approach is to first determine the range of layers where time information is concentrated based on readability, and then determine the target intervention layer within that range based on coordinate compatibility. Specifically, if the readable layers are concentrated in deeper layer intervals, select layers within those intervals; if they are concentrated in shallower single-layer positions, use those layers as candidate layers; then, filter by the coordinate compatibility of each layer, and determine the layers whose coordinate compatibility meets the preset compatibility criteria as the target intervention layers. For example, the layer whose time coordinate migration prediction is worse than the reference prediction result can be determined as the target intervention layer. This layer selection principle can be applied across models.

[0234] Another alternative approach is to integrate readability and coordinate compatibility into a unified selection criterion. Specifically, the readability of time information and coordinate compatibility can be considered simultaneously for each candidate layer. For example, the two can be combined according to a preset emphasis, and the layer with the best overall performance can be determined as the target intervention layer.

[0235] In addition, the target intervention layer can be a layer set consisting of one or more layers; after it is determined, the layer selection results can be reviewed, and if the review is not ideal, the combination method can be adjusted and the layer can be re-determined.

[0236] In an optional embodiment, if the target multimodal data contains only a single modality, the coordinate compatibility can be determined according to a preset default value, and the target intervention layer can be determined only based on readability, so that layer selection processing can still be implemented in the case of a single modality.

[0237] After determining the target intervention layer, the corresponding time interval for the target object can be determined based on this and the initial time interval. In practical applications, determining the corresponding time interval for the target object in the target multimodal data based on the target intervention layer and the initial time interval can also be achieved through various methods.

[0238] One alternative approach is to implement intervention at the target intervention layer and combine the results obtained from the intervention with the initial time interval. Specifically, intervention is implemented based on the target intervention layer, allowing post-intervention processing to utilize the time information carried by the latent state of this layer to compare and integrate the intervention results with the initial time interval as a benchmark. Consistent parts are directly retained, while discrepancies are adjusted according to the intervention results. The form of intervention can be determined based on the characteristics of the target intervention layer, which will not be elaborated here.

[0239] Another alternative approach is to intervene at the target intervention layer during the model's inference process. Specifically, when inference reaches the target intervention layer, intervention is implemented using time-related information in the hidden state of that layer. The model continues inference based on the intervened hidden state and outputs the time location result. This result is then compared with the initial time interval to determine the time interval corresponding to the target object.

[0240] In addition, interventions can be limited to the time frame relevant to the target object to reduce the impact on other outputs; when interventions fail to yield effective results, the time frame corresponding to the target object can be determined directly based on the initial time frame.

[0241] In an alternative embodiment, behavioral-level diagnostics can be performed on the model before determining the target intervention layer to determine whether it is worthwhile to enable mechanistic-level intervention.

[0242] Specifically, behavioral-level diagnostics are performed solely based on the model's output, without accessing the model's internal state. They can be based on labeled evaluation samples to offline statistically analyze failure patterns in the model's time-localization output. This includes metrics such as the directional bias of the predicted time interval center relative to the true center, the proportion of outputs with premature clustering, and the proportion of outputs that cannot be parsed into valid time intervals. Based on these metrics, it can be determined whether the model exhibits a directional time bias in the target multimodal data scenario. For example, if a model predicts the center of the interval less than one unit of time ahead without rendering time coordinates, but after introducing separate time coordinates, the predicted center becomes significantly earlier, and the proportion of premature clustering increases significantly, it can be determined that the model has a directional time bias. Behavioral-level diagnostics can serve as an offline basis for deciding whether to initiate further intervention. When no significant time bias is present, the time interval corresponding to the target object can be directly determined based on the initial time interval, reducing unnecessary processing overhead.

[0243] In the embodiments of this specification, by acquiring the hidden states of each word during the reasoning process, the readability of time information in the hidden states and the coordinate compatibility between different modalities are determined. Based on the readability and coordinate compatibility, the target intervention layer is determined, and the time interval corresponding to the target object in the target multimodal data is determined by combining the initial time interval. This allows the time localization processing to go from the output level to the model mechanism level. Based on the distribution of time information within the model and the availability across modalities, the position of the intervention processing is determined, providing a clear target for the targeted correction of the localization results and improving the targeting of the localization correction processing.

[0244] In one optional embodiment of this specification, determining the readability of the target object's temporal information in the latent state, and the coordinate compatibility of the temporal information among the various modalities of the target multimodal data, includes: Based on the hidden state of the lexical units corresponding to the time coordinate, time prediction probes are trained on each layer of the decoder of the multimodal data processing model. Based on the prediction accuracy of time information by the time prediction probe at each layer, the readability of the target object's time information in the hidden state is determined. The time prediction probe is applied to the hidden state of a word in a different modality than the word corresponding to the time coordinate to obtain the migration prediction result; Based on the migration prediction results and the preset reference prediction results, the degree of coordinate compatibility of time information among the modes of the target multimodal data is determined.

[0245] After obtaining the hidden state of each word, time prediction probes can be trained on each layer of the decoder based on the hidden state of the word corresponding to the time coordinate, in order to determine the readability of the target object's time information in the hidden state, as well as the coordinate compatibility of the time information between the modalities of the target multimodal data.

[0246] The decoder is the component in a multimodal data processing model that performs output-side transformation processing. The model is organized in layers, with each layer of the decoder sequentially transforming words, and the layer number increasing from the input side to the output side. Specifically, after transformation at each layer, words form hidden states in the corresponding layers. Furthermore, the readability of temporal information in the hidden states of different layers can vary. Therefore, temporal prediction probes are trained on each layer of the decoder to examine the readability of temporal information layer by layer.

[0247] A time prediction probe is a probe used to predict time information from the hidden state of a word. It is a lightweight regression model trained separately on each layer of the decoder. Specifically, the time prediction probe uses the hidden state of the word corresponding to the time coordinate as the regression input and the normalized time corresponding to the word as the prediction target. Each layer trains its own time prediction probe, and the time prediction probes of different layers do not share parameters. The training labels use normalized time, for example, it can be calculated using the formula y = t / T, where y is the normalized time, t is the media time corresponding to the word, and T is the duration of the target multimodal data. Specifically, after normalizing the media time according to the duration of the target multimodal data, the time labels of different target multimodal data are on the same scale and can be used together for training.

[0248] Furthermore, the time prediction probe can be a lightweight regression model, such as a linear mapping regression model, which directly outputs the predicted value of normalized time after inputting the hidden state; the time prediction probes of each layer are independent of each other, and their prediction performance can reflect the situation of the hidden state of the corresponding layer carrying time information.

[0249] Prediction accuracy is a metric used to measure the ability of a time prediction probe to predict temporal information. Specifically, prediction accuracy characterizes how close the temporal information read from the latent state by the probe is to the actual temporal information of the lexical units. For example, the coefficient of determination can be used as a metric. The coefficient of determination characterizes the degree to which the time prediction probe interprets the temporal information, and its value does not exceed one. The closer the value is to one, the more linearly the temporal information can be read. Furthermore, when the temporal information exists in the latent state in a linear form, simple regression can predict with high accuracy; when it exists in a complex and dispersed form, lightweight regression has lower prediction accuracy.

[0250] Transfer prediction results are obtained by applying a time prediction probe to the hidden state of a word corresponding to another modality. Specifically, a time prediction probe trained at a certain layer is directly applied to the hidden state of a word corresponding to another modality without any adaptation or retraining to obtain the transfer prediction result. Furthermore, the word corresponding to the other modality can be a word from a different modality than the word corresponding to the time coordinate, such as a word corresponding to the audio portion of media. The transfer prediction result is used to verify whether the readout method of the time coordinate can be applied to other modalities without adjustment.

[0251] The reference prediction result is a pre-set baseline prediction result, serving as a benchmark for judging the quality of the migration prediction result. Specifically, the reference prediction result can be a prediction result obtained solely based on the statistical values ​​of the time information of corresponding words in another modality. For example, the statistical mean of the normalized time of corresponding words in another modality can be used as a unified prediction for all words, serving as the simplest baseline prediction. Furthermore, if the migration prediction result is worse than this baseline, it indicates that the time coordinate reading method cannot be shared across modalities, i.e., negative migration has occurred.

[0252] In practical applications, determining the readability of the target object's temporal information in its hidden state, as well as the coordinate compatibility of the temporal information across different modalities of the target multimodal data, can be achieved through various methods. For example, one approach is to first train temporal prediction probes at each layer of the decoder based on the hidden state of the tokens corresponding to the temporal coordinates.

[0253] One alternative approach is to independently fit time prediction probes layer by layer on each layer of the decoder. Specifically, for each layer, the hidden state of the term corresponding to the time coordinate in that layer is used as the regression input, and the normalized time corresponding to the term is used as the prediction target. Time prediction probes are independently fitted on that layer, and the fitting of each layer does not affect the others. Through independent fitting layer by layer, the prediction performance of each layer's probe can reflect the temporal information carried by the hidden state of the corresponding layer.

[0254] Another alternative approach is to perform offline training based on a pre-defined sample set. Specifically, the sample set consists of multiple multimodal data sets with time coordinates, divided into discovery and development sets, etc. The time prediction probe is fitted on the discovery set, and the parameter determination logic for each layer remains consistent. Training is completed offline before formal inference, and the trained time prediction probe can be used directly during the inference phase.

[0255] Another alternative approach is to organize the time-corresponding words from multiple multimodal datasets with time coordinates into training samples, which can then be used to train time prediction probes at each layer. Specifically, each probe is fitted based on this set of training samples, differing only in the model layer from which the input hidden states are taken. By sharing training samples, the impact of differences in training data on the comparison of results between layers can be reduced.

[0256] Furthermore, the parameters of the multimodal data processing model can be kept constant during the training of the prediction probe, and fitting can be completed using only the hidden states read during inference.

[0257] After training the time prediction probe layer by layer, the readability of the target object's time information in the hidden state can be determined based on the prediction accuracy of the time prediction probe for time information in each layer.

[0258] One alternative approach is to organize the prediction accuracy of the time prediction probes at each layer of the decoder by layer, obtaining a curve showing the change in prediction accuracy with layer number. Based on this curve, the readability of time information in the hidden states of each layer can be determined. Specifically, layers with higher prediction accuracy correspond to layers with higher readability, and layers with higher readability indicate a higher degree to which time information is linearly read out, thus obtaining the distribution of readability with layer number.

[0259] Another alternative approach is to compare the prediction accuracy of the temporal prediction probes at each layer of the decoder to determine layers or layer intervals with higher readability. Specifically, layers with relatively high prediction accuracy are identified as layers with higher readability, and adjacent layers with high prediction accuracy can constitute layer intervals with higher readability. By comparing the prediction accuracy between layers, readability can be determined at the layer granularity.

[0260] In addition, readability can be determined by combining linear readout characteristics. For example, when the prediction accuracy of each layer is comparable, the layer that can read out the time information in a simpler way is determined to be the layer with higher readability.

[0261] After determining the readability, the time prediction probe can be applied to the hidden state of a word in a different modality than the word corresponding to the time coordinate to obtain the migration prediction result.

[0262] One alternative approach is to directly apply the time prediction probe trained at one layer to the hidden state of the corresponding word in another modality. Specifically, the probe is applied as is to the hidden state of the corresponding word in that layer of another modality without any adaptation or retraining. The prediction result output by the probe is the transfer prediction result, which can be used to verify the applicability of the time coordinate readout method to the hidden state of word in other modalities.

[0263] Another alternative approach is to apply the trained temporal prediction probes of each layer to the hidden states of corresponding lexical units in another modality at the corresponding layer, obtaining layer-by-layer transfer prediction results. Specifically, each layer probe outputs its prediction results and records them layer by layer for subsequent layer-by-layer comparisons. Through layer-by-layer transfer prediction, the cross-modal availability of temporal information readout methods at each layer can be characterized.

[0264] Another alternative approach is to apply temporal prediction probes to multiple lexical units of another modality to obtain lexical-level transfer prediction results. Specifically, the other modality can correspond to multiple lexical units, such as multiple lexical units corresponding to the audio portion of media. The temporal prediction probes are applied to the latent states of each lexical unit to obtain multiple corresponding prediction results. By performing transfer prediction on lexical units, the prediction performance on different lexical units can be compared.

[0265] Furthermore, migration prediction can be performed separately on multiple multimodal datasets with time coordinates to obtain migration prediction results covering multiple samples.

[0266] After obtaining the migration prediction results, the degree of coordinate compatibility of time information among the modes of the target multimodal data can be determined based on the migration prediction results and the preset reference prediction results.

[0267] One alternative approach is to compare the migration prediction error with the error of the reference prediction result to determine the degree of coordinate compatibility. Specifically, the error between the migration prediction result and the actual time information of the word, as well as the error between the reference prediction result and that time information, are calculated separately. If the migration prediction error is worse than the error of the reference prediction result, it indicates that the time coordinate readout method has not gained any gain after migrating to another modality, and the coordinate compatibility is determined to be incompatible; otherwise, it is determined to be compatible.

[0268] Another alternative approach is to determine the degree of coordinate compatibility based on the relative advantage of the migration prediction results over the reference prediction results. Specifically, the smaller the error of the migration prediction results relative to the reference prediction results, the more usable the time coordinate readout method is in the other mode, and the higher the degree of compatibility; for example, when the errors are similar, it is considered partially compatible, and when the error is significantly smaller, it is considered to have a higher degree of compatibility.

[0269] Another alternative method is to compare the migration prediction results with the reference prediction results at multiple layers to comprehensively determine the coordinate compatibility. Specifically, the migration prediction error of each layer is compared with the error of the reference prediction result one by one. Layers with migration prediction errors worse than reference prediction result errors are identified. When fewer layers exhibit this condition, the compatibility is higher; when more layers exhibit this condition, the compatibility is lower. The coordinate compatibility is then comprehensively determined based on this.

[0270] In addition, the comparison results of migration prediction error and reference prediction result error can be summarized on multiple samples, and the degree of coordinate compatibility can be determined based on the summarized results, thereby improving the stability of the judgment.

[0271] In an alternative embodiment, the layer selection principle, rather than a specific layer number, can also be used as a transferable processing experience.

[0272] Specifically, the location of the time information readability layer is related to the model architecture. For example, the time information readability layer of one model may be located in a deeper layer range, while the time information readability layer of another model may be located in a shallower layer range. What can be transferred between models is the principle of selecting the target intervention layer based on readability criteria and transfer criteria. For example, the layer range with high readability of time information, worse transfer prediction of time coordinates than the reference prediction result, and better transfer prediction of time coordinates corresponding to attached presentation conditions than the reference prediction result can be used as the candidate range for the target intervention layer.

[0273] In an alternative embodiment, the model parameters, data, and evaluation samples can be frozen into a discovery set, development set, validation set, and final test set before analysis.

[0274] Specifically, the time prediction probe is fitted only on the discovery set, the development set is used to determine the intervention form and intensity, and the validation set and final test set are only used for evaluation and do not participate in the determination of the time prediction probe and intervention-related parameters.

[0275] In the embodiments of this specification, time prediction probes are trained on each layer of the decoder based on the hidden states of the lexical units corresponding to the time coordinates. The readability of time information in the hidden states is determined based on the prediction accuracy of each layer's probes. The time prediction probes are then applied to the hidden states of lexical units corresponding to another modality. Combined with preset reference prediction results, the coordinate compatibility of time information between modalities is determined. This distinguishes between two situations: whether time information exists in the hidden states and whether the time coordinate reading method can be shared across modalities. Readability and coordinate compatibility are independent of each other, which can avoid misjudging readable performance within a single modality as shared performance across modalities. This provides a mechanistic basis for determining the target intervention layer, enabling the determination of the target intervention layer to be carried out in combination with the distribution of time information within the model and its availability across modalities, thereby enhancing the targeting of intervention processing.

[0276] In one optional embodiment of this specification, determining the time interval corresponding to the target object in the target multimodal data based on the target intervention layer and the initial time interval includes: Obtain the anchor prediction result and the corresponding anchor hidden state. The anchor prediction result is the prediction result obtained by inference of the target multimodal data under rendering without superimposed time coordinates. The anchor hidden state is the hidden state of the word corresponding to the time coordinates aligned frame by frame during the process of inference to obtain the anchor prediction result. At the target intervention layer, the hidden state of the word corresponding to the time coordinate is oriented and interpolated along the direction of the hidden state of the anchor point, and reasoning is performed based on the interpolated hidden state to obtain the repair time interval. Based on the repair time interval and the initial time interval, the time interval corresponding to the target object in the target multimodal data is determined.

[0277] After determining the target intervention layer, intervention and repair processing can be performed based on the target intervention layer and the initial time interval. That is, with the inference result and hidden state under the rendering without superimposed time coordinates as reference, directional interpolation is performed on the hidden state of the word corresponding to the time coordinate on the target intervention layer. The repair time interval is obtained based on the inference of the interpolated hidden state, and then combined with the initial time interval to determine the time interval corresponding to the target object in the target multimodal data.

[0278] Anchor point prediction results are obtained by inferring the target multimodal data under rendering without overlaying time coordinates. Specifically, rendering without overlaying time coordinates can be, for example, rendering a pure black screen, without displaying any time coordinates or content information. When the rendered data is input into the multimodal data processing model for inference, the inference process does not involve reading or utilizing time coordinates, and the obtained anchor point prediction results do not contain the influence introduced by time coordinates. Furthermore, anchor point prediction results can serve as an independent estimate of the target object's temporal position. Rendering without overlaying time coordinates is one optional input format for obtaining anchor point prediction results; in an optional embodiment, anchor point prediction results can also be obtained by inferring other input formats different from multimodal data with time coordinates, such as inputting audio data from the target multimodal data into the multimodal data processing model in audio format for inference.

[0279] Furthermore, when the target multimodal data is audio data, the anchor prediction result can be composed of multiple inference results. Specifically, the audio data can be rendered as a reference visual image without time coordinate information, such as a pure black screen. The reference visual image is then input into the multimodal data processing model for inference to obtain the prediction result in the visual rendering form. Alternatively, the audio data can be directly input into the multimodal data processing model in audio form for inference to obtain the prediction result in the direct audio form. None of the inferences that constitute the anchor prediction result include the influence of time coordinates.

[0280] Anchor point hidden states are the frame-aligned hidden states of the words corresponding to the time coordinates, generated by the multimodal data processing model during the inference process to obtain anchor point prediction results. Specifically, for the same target multimodal data, the frame structure and word arrangement are the same in rendering without and with time coordinates, only the content presented in the image differs. Therefore, during the inference process to obtain anchor point prediction results, the hidden states generated by the model at the frame-aligned positions of the words corresponding to the time coordinates can correspond one-to-one with the hidden states of the words corresponding to the time coordinates. Furthermore, anchor point hidden states can serve as reference directions for directional interpolation.

[0281] Furthermore, when the anchor prediction result is composed of multiple inference paths, the anchor hidden state is taken from the hidden state corresponding to the rendering inference path with a visual frame structure. Specifically, the inference path in the form of direct audio transmission does not produce the word arrangement corresponding to the image frame structure, and its hidden state is difficult to align frame by frame; the hidden state produced by the rendering inference path with a visual frame structure maintains the same frame structure as the rendering overlaid with the time coordinates, and can be aligned according to the hidden state of the word corresponding to the frame and time coordinates. Therefore, the hidden state of this inference path is taken as the anchor hidden state.

[0282] The repaired time interval is the time interval corresponding to the target object in the target multimodal data, obtained through inference based on the interpolated hidden states. Specifically, after directional interpolation is performed at the target intervention layer, the interpolated hidden states are used as input for subsequent inference to continue the inference process. The time interval corresponding to the target object is obtained from the inference result, which is the repaired time interval. Furthermore, the repaired time interval can be regarded as the localization result obtained by re-inference after directional intervention in the inference process that obtained the initial time interval.

[0283] Performing intervention and repair requires anchor point prediction results and anchor point hidden states as references. In practical applications, obtaining anchor point prediction results and their corresponding anchor point hidden states can be achieved through various methods.

[0284] One alternative approach is to perform inference on the target multimodal data without overlaying time coordinates during rendering, obtaining the anchor point prediction result and the anchor point hidden state simultaneously. Specifically, the rendering can be replaced with a pure black screen, maintaining the frame structure and word arrangement. The rendered data is then input into a multimodal data processing model for inference to obtain the anchor point prediction result. When the inference reaches the target intervention layer, the hidden state of the words corresponding to the time coordinates at that layer, aligned frame-by-frame, is read as the anchor point hidden state. By obtaining both the anchor point prediction result and the anchor point hidden state simultaneously in a single inference, the consistency of their origins can be maintained.

[0285] Another alternative approach, when the target multimodal data is audio data, is to obtain anchor prediction results and anchor hidden states through multi-path inference. Specifically, the audio data is rendered as a contrasting visual image without temporal coordinate information, such as a pure black screen, and input into the multimodal data processing model for inference to obtain the prediction result in the visual rendering form. The audio data is directly input into the multimodal data processing model in audio form for inference to obtain the prediction result in the direct audio form. The results of all inference paths are used together as the anchor prediction result, and when the inference of the visual rendering form reaches the target intervention layer, the hidden states of the words corresponding to the time coordinates, aligned frame by frame, are read as the anchor hidden states. By constructing the anchor prediction result through multi-path inference, a comprehensive reference for inference of multiple forms can be provided for intervention and repair.

[0286] Another alternative approach is to acquire the anchor prediction results and anchor hidden states in stages. First, inference is performed to obtain the anchor prediction results, and then the anchor hidden states are acquired as needed. Specifically, a full inference is performed first under rendering without overlaying time coordinates to obtain the anchor prediction results. Before performing directional interpolation, inference is performed again for the same rendering input. When the inference reaches the target intervention layer, the hidden states of the terms corresponding to the time coordinates, aligned frame-by-frame, are read as the anchor hidden states. Both inferences use the same model parameters and input data, and the anchor hidden states can be used in conjunction with the anchor prediction results. By acquiring the hidden states as needed, the overhead of acquiring them can be omitted when intervention repair is not required.

[0287] In addition, the frame alignment information of the anchor point hidden state can be recorded when obtaining the anchor point prediction results. For example, the frame position and word position corresponding to each anchor point hidden state can be recorded, so that the hidden state of the word corresponding to the time coordinate can be matched one by one during directional interpolation.

[0288] After obtaining the anchor point prediction results and anchor point hidden states, directional interpolation can be further performed. In practical applications, at the target intervention layer, the hidden states of the lexical units corresponding to the time coordinates are directionally interpolated along the direction of the anchor point hidden states, and inference is performed based on the interpolated hidden states to obtain the repair time interval. This can also be achieved through various methods.

[0289] One alternative approach is to perform directional interpolation on a per-terminal basis for each lexical unit corresponding to the time coordinate. Specifically, for lexical unit i corresponding to the time coordinate, obtain the hidden state of lexical unit i in the target intervention layer, and the anchor point hidden state aligned with that lexical unit frame by frame. The difference between the two is the direction of state deviation caused by the time coordinate. For example, directional interpolation can be performed on the hidden state of lexical unit i according to the following formula:

[0290] in, The lexical units corresponding to the time coordinates on the target intervention layer The hidden state, For the word element Hidden state of anchor points aligned to frames For interpolation intensity, It represents the direction of state deviation caused by the time coordinate. The meaning of this formula is to... The hidden state moves along the difference between the anchor hidden state and the word hidden state according to the interpolation intensity. When the interpolation intensity is zero, the hidden state does not move, that is, no intervention is implemented; when the interpolation intensity is one, the interpolated hidden state is consistent with the anchor hidden state.

[0291] After completing the directional interpolation, the interpolated hidden state is used as the input for subsequent inference in the target intervention layer to continue inference. The time interval corresponding to the target object is obtained from the inference result and used as the repair time interval.

[0292] Another alternative approach is to perform directional interpolation uniformly on all lexical units corresponding to the time coordinates. Specifically, the hidden states of anchor points and the hidden states of lexical units corresponding to the time coordinates are aligned frame-by-frame and correspond one-to-one. All hidden states of both can be organized into aligned hidden state sets. Interpolation calculations are then performed uniformly on both sets at the target intervention layer to obtain the interpolated hidden states of all lexical units at once. Further reasoning is then used to obtain the repair time interval. By uniformly performing interpolation, the hidden states of each lexical unit can maintain a consistent adjustment direction.

[0293] Another alternative approach is to predetermine the interpolation strength and then perform directional interpolation according to the determined interpolation strength. Specifically, the interpolation strength controls the proportion by which the hidden state moves towards the anchor hidden state. For example, the interpolation strength can be determined by scanning the discovery set and development set during the offline phase, or it can be determined from candidate values ​​such as 0.5 or 0.75. After determining the interpolation strength, directional interpolation is performed on the target intervention layer according to the determined interpolation strength, and inference continues based on the interpolated hidden state to obtain the repair time interval.

[0294] In addition, the above methods can be combined. For example, the interpolation intensity can be determined first, and then directional interpolation can be performed according to the determined interpolation intensity when the inference reaches the target intervention layer to obtain the repair time interval.

[0295] After obtaining the repair time interval, the corresponding time interval of the target object in the target multimodal data can be determined by combining it with the initial time interval. In practical applications, determining the corresponding time interval of the target object in the target multimodal data based on the repair time interval and the initial time interval can also be achieved through various methods.

[0296] One alternative approach is to use both the repair time interval and the initial time interval as candidate time intervals, and combine them with other reference information to determine the corresponding time interval of the target object in the target multimodal data. Specifically, the initial time interval is the positioning result obtained through inference without intervention, and can be used as a baseline candidate without intervention; the repair time interval is the positioning result obtained through inference after directional interpolation, and can be used as a repair candidate after intervention. During the determination, other reference information related to the target object's temporal location can be combined, such as anchor point prediction results or estimation results obtained from other sources, to perform the determination process on both candidate time intervals. By incorporating the positioning results before and after intervention into the determination process, comprehensive utilization of both sets of positioning results can be achieved.

[0297] Another alternative approach is to first determine the consistency between the repair time interval and the initial time interval, and then determine the corresponding time interval of the target object in the target multimodal data based on the determination result. Specifically, if the two are consistent, the time interval can be directly determined as the time interval corresponding to the target object; if the two are inconsistent, other reference information can be combined to perform determination processing from the repair time interval and the initial time interval to obtain the corresponding time interval of the target object in the target multimodal data.

[0298] In addition, other processing methods can be adopted to determine the corresponding time interval of the target object by simultaneously using the repair time interval and the initial time interval, or the above methods can be combined to adapt to the determination requirements of different scenarios.

[0299] In an alternative embodiment, directional interpolation can be applied only to the lexical units corresponding to the time coordinates, without interfering with lexical units carrying other content.

[0300] Specifically, by limiting the directional interpolation to the range of words corresponding to the time coordinate, the hidden state of the words corresponding to the time coordinate can be adjusted. Words carrying other content do not undergo directional interpolation, their hidden states remain unchanged, and the model's processing of evidence content is not disturbed.

[0301] In an alternative embodiment, the aforementioned directional interpolation can also be performed via the hook mechanism of the model inference framework.

[0302] Specifically, hooks are registered at the target intervention layer. When the inference reaches the target intervention layer, the hooks are triggered. The hidden state of the term corresponding to the time coordinate is read from the inference process. After performing directional interpolation, the interpolated hidden state is written back to the inference process. The model continues to infer based on the interpolated hidden state according to the original process to obtain the repair time interval. The intervention process does not modify the model's weights and structure. After the intervention is completed, the original inference process is restored.

[0303] In an optional embodiment, the source of the anchor prediction result and the anchor hidden state can be replaced by rendering without superimposed time coordinates with attached rendering. Specifically, when the target multimodal data adopts a separate presentation condition and needs to compensate for the time positioning deviation by intervention repair, the frame-aligned hidden state obtained by rendering the same sample according to the spectrogram with time coordinates can be used as the donor direction. On the target intervention layer, the hidden state of the word corresponding to the time coordinate is interpolated according to the aforementioned directional interpolation method, so that the intervention changes from weakening the state offset introduced by the time coordinate to introducing the time information available under the attached presentation condition; accordingly, the anchor prediction result can be obtained based on the inference result of the attached rendering, and the anchor hidden state is taken as the frame-aligned hidden state of the word corresponding to the time coordinate in this inference.

[0304] Accordingly, in an optional embodiment of this specification, the anchor point hidden state is the hidden state of the word corresponding to the time coordinate aligned frame by frame during the process of reasoning to obtain the anchor point prediction result; wherein, the anchor point prediction result is the prediction result obtained by reasoning on the target multimodal data under the rendering of the time coordinate superimposed and the time coordinate and the evidence content located in the same visual data.

[0305] In the embodiments of this specification, by taking both the anchor point prediction result and the anchor point hidden state from the reasoning process under the rendering of the time coordinate and the evidence content in the same visual data, the direction of directional interpolation is changed from weakening the state offset introduced by the time coordinate to introducing the time information available under the attached presentation conditions, so as to provide a reference that is compatible with the attached presentation conditions for repairing the time interval.

[0306] In the embodiments of this specification, the target multimodal data is inferred under rendering without superimposed time coordinates to obtain anchor point prediction results and anchor point hidden states that do not contain the influence of time coordinates. On the target intervention layer, directional interpolation is performed on the hidden states of the words corresponding to the time coordinates, and the repair time interval is obtained by re-inferring based on the interpolated hidden states, so that the repaired positioning results are free from the directional interference introduced by the time coordinates. The repair time interval and the initial time interval are used together as candidate results to provide two sets of positioning results before and after intervention for the final determination of processing, thereby improving the usability of time positioning results.

[0307] In one optional embodiment of this specification, the hidden state of the lexical corresponding to the time coordinate is oriented and interpolated along the direction of the hidden state of the anchor point, including: Based on the deviation direction and magnitude between the initial time interval and the anchor point prediction results, the target interpolation intensity of the directional interpolation is determined. Based on the target interpolation intensity, the hidden state of the lexical corresponding to the time coordinate is oriented and interpolated along the direction of the hidden state of the anchor point.

[0308] After obtaining the anchor point prediction results and anchor point hidden states, directional interpolation can be performed on the hidden states of the words corresponding to the time coordinates to reduce the influence of the time coordinates on the hidden states. The interpolation intensity used for directional interpolation can be adaptively determined for the current sample. The process may include: determining the target interpolation intensity based on the deviation direction and deviation magnitude between the initial time interval and the anchor point prediction results; and performing directional interpolation on the hidden states of the words corresponding to the time coordinates along the direction of the anchor point hidden states according to the target interpolation intensity.

[0309] The meaning of interpolation intensity is described in the foregoing embodiments and will not be repeated here.

[0310] In different samples, the temporal position of the target object in the target multimodal data varies, and the degree of deviation of the time coordinates from the latent state of each sample also differs. Performing directional interpolation with a uniform fixed intensity may be beneficial for samples where the target object appears at a mid-to-late time, but may overcorrect samples where the target object actually appears earlier. Furthermore, the temporal position of the target object is unknown during the inference stage, making it difficult to pre-select an appropriate interpolation intensity for each sample. Based on the deviation between the initial time interval and the anchor point prediction results, determining the target interpolation intensity for each sample according to its own deviation signal can match the intervention intensity with the degree of deviation of each sample, reducing the possibility of overcorrection.

[0311] In practical applications, the target interpolation intensity of directional interpolation can be determined based on the deviation direction and magnitude between the initial time interval and the anchor point prediction result, which can be achieved through various methods.

[0312] One possible approach is to first determine the direction and magnitude of the deviation, and then determine the target interpolation strength based on the positive correlation between the deviation magnitude and the target interpolation strength. Specifically, the anchor prediction result includes an estimate of the time position corresponding to the target object. The time positions corresponding to the two sets of results can be represented by the center position of the initial time interval and the center position of the corresponding time position in the anchor prediction result, respectively. The difference between the two center positions represents the deviation between the initial time interval and the anchor prediction result, and the direction of the difference is the deviation direction. For example, the center of the initial time interval can be earlier or later than the center of the corresponding time position in the anchor prediction result. The magnitude of the difference is the deviation magnitude, which can be measured in time units or normalized. After determining the deviation direction and magnitude, the target interpolation strength can be determined according to the positive correlation mapping relationship; the larger the deviation magnitude, the greater the target interpolation strength. Determining the target interpolation strength through positive correlation mapping allows the intervention strength to increase with the deviation magnitude, matching the degree of deviation in the sample.

[0313] Another alternative method is to determine the target interpolation intensity using a piecewise mapping approach. Specifically, the deviation direction and magnitude can be determined as described in the previous alternative method, and will not be repeated here. After determining the deviation magnitude, the range of deviation magnitude can be divided into several intervals, and a corresponding interpolation intensity level can be set for each interval. Intervals with larger deviation magnitudes correspond to higher intensity levels, and intervals with smaller deviation magnitudes correspond to lower intensity levels. After determining the interval in which the deviation magnitude falls, the intensity level corresponding to that interval is determined as the target interpolation intensity. By determining the target interpolation intensity through piecewise mapping, the deviation magnitude of the sample can be approximately matched with discrete intensity levels, making the determination process of the target interpolation intensity simple and controllable.

[0314] Another alternative method is to use linear mapping to determine the target interpolation intensity, thereby obtaining continuous intensity values. Specifically, the deviation direction and deviation magnitude can be determined as described in the aforementioned alternative methods. After determining the deviation magnitude, the deviation magnitude can be mapped to a continuous target interpolation intensity according to a pre-set linear mapping relationship. The mapping relationship can be set such that the larger the deviation magnitude, the larger the target interpolation intensity, and the range of values ​​for the mapped output can be set according to actual needs. Obtaining continuous target interpolation intensity through linear mapping allows the target interpolation intensity to change smoothly with the deviation magnitude, providing more refined intensity values ​​for samples with varying degrees of deviation.

[0315] In addition, the above-mentioned mapping methods can be combined. For example, piecewise mapping or linear mapping can be used for different ranges of deviation, or the mapping relationship can be adaptively adjusted according to the direction of deviation.

[0316] All of the above-mentioned optional methods can determine the target interpolation strength based on the deviation of the current sample itself. After obtaining the target interpolation strength, directional interpolation can be performed according to the target interpolation strength. In practical applications, directional interpolation can be performed on the hidden state of the lexical corresponding to the time coordinate along the direction of the anchor point's hidden state according to the target interpolation strength, which can also be achieved through various methods.

[0317] An alternative method is to perform directional interpolation as described in the foregoing embodiments, using the target interpolation intensity as the intensity employed in the directional interpolation. Specifically, the foregoing embodiments describe an interpolation method where the hidden state moves along the direction of the anchor point's hidden state. The interpolation calculation process is consistent with that described in the foregoing embodiments and will not be repeated here. During execution, for the current sample, when inference reaches the target intervention layer, the target interpolation intensity is determined based on the deviation direction and magnitude between the initial time interval and the anchor point prediction result. Subsequently, following the foregoing interpolation method, the hidden state of the term corresponding to the time coordinate is moved along the direction of the anchor point's hidden state by the proportion indicated by the target interpolation intensity to obtain the interpolated hidden state. Inference is then continued based on the interpolated hidden state to obtain the repair time interval. By reusing the interpolation method in the foregoing embodiments, only the interpolation intensity used needs to be replaced with the target interpolation intensity to adapt the repair processing to the deviation of the current sample.

[0318] Another alternative approach is to organize the determination of the target interpolation strength and the execution of directional interpolation as consecutive processing steps in the inference process, performed sequentially for each sample. Specifically, the multimodal data processing model performs inference based on multimodal data with time coordinates. When inference reaches the target intervention layer, it first reads the latent state of the term corresponding to the time coordinate, then combines the initial time interval of the current sample with the anchor prediction result to determine the deviation direction and magnitude, thereby determining the target interpolation strength. Subsequently, according to the target interpolation strength, directional interpolation is performed on the latent state of the term corresponding to the time coordinate along the direction of the anchor latent state. The interpolated latent state is then used as input for subsequent inference at the target intervention layer, and inference continues, obtaining the repair time interval from the inference result. By completing the strength determination and interpolation execution within the inference process, the processing between samples is independent, and each sample obtains the intervention strength corresponding to its deviation.

[0319] Another alternative approach is to determine the semantic meaning of directional interpolation based on the deviation direction, and then perform directional interpolation on each word corresponding to the time coordinate. Specifically, the deviation direction is used to distinguish the deviation pattern of the sample. For example, it can be an early deviation (the initial time interval is earlier than the predicted time position of the anchor point) or a late deviation (it is relatively delayed). For early deviations, moving the latent state of the word along the direction of the anchor point's latent state can suppress the early bias introduced by the time coordinate. Combining the target interpolation intensity determined by the deviation direction, the intervention intensity can be matched with the deviation pattern of the sample. The target interpolation intensity of the current sample can be uniformly applied to each word corresponding to the time coordinate, with each word moving along the direction of its respective frame-aligned anchor point latent state by the same proportion.

[0320] In addition, the above methods can be combined. For example, the target interpolation intensity of the current sample can be determined first according to the aforementioned optional methods, and then directional interpolation can be performed when the inference reaches the target intervention layer. Based on the interpolated hidden state, the inference can continue to obtain the repair time interval.

[0321] In an alternative embodiment, a candidate range of interpolation intensity can be determined during the offline phase, and then the target interpolation intensity can be determined sample by sample within the candidate range.

[0322] Specifically, during the offline phase, a fixed candidate grid of interpolation intensity can be scanned according to the discovery set and the development set. For example, multiple discrete intensity candidates can be scanned, and the candidate values ​​can be two intensities, such as 0.5 and 0.75. The fixed intensity or intensity range to be used for deployment can be determined by the development set. When the target interpolation intensity is determined on a sample-by-sample basis, the target interpolation intensity of each sample can be determined within the determined intensity range, based on the deviation direction and deviation magnitude of the sample according to the aforementioned optional methods.

[0323] In an alternative embodiment, the target interpolation intensity can also be limited to a preset range.

[0324] Specifically, a range of allowed values ​​can be set for the target interpolation intensity. When the target interpolation intensity determined for each sample exceeds the upper limit of the range, the upper limit value is used; when it is below the lower limit of the range, the lower limit value is used. This controls the intervention intensity within a reasonable range and avoids over-correction of samples with special deviations from the morphology.

[0325] In the embodiments of this specification, the target interpolation intensity of directional interpolation is determined sample by sample based on the deviation direction and magnitude between the initial time interval and the anchor point prediction result. According to the target interpolation intensity, the latent state of the term corresponding to the time coordinate is directionally interpolated along the direction of the anchor point's latent state. This allows the intervention intensity to adapt to the deviation characteristics of different samples, with samples having larger deviation magnitudes receiving stronger repair interventions and samples having smaller deviation magnitudes receiving weaker repair interventions. This reduces the over-correction of some samples by fixed-intensity repair and improves the adaptability of the repair processing to sample differences. The directional interpolation is performed along the direction of the anchor point's latent state and coordinates with the deviation direction, suppressing premature bias for premature deviations. This provides repair results adapted to the sample deviation situation for determining the time interval corresponding to the target object in the target multimodal data based on the initial time interval.

[0326] In one optional embodiment of this specification, determining the time interval corresponding to the target object in the target multimodal data based on the repair time interval and the initial time interval includes: Calculate the positional differences between the initial time interval and the repair time interval and the prediction results of at least one anchor point, respectively; The time interval with the smallest positional difference is determined as the time interval corresponding to the target object; When the location differences are the same, the initial time interval is determined as the time interval corresponding to the target object; If the prediction results of at least one anchor point cannot be resolved, the initial time interval is determined to be the time interval corresponding to the target object.

[0327] After obtaining the repair time interval and the initial time interval, a process can be performed to determine the time interval corresponding to the target object in the target multimodal data from the two. This process uses the anchor prediction result as a reference and determines the time interval that is closer to the time position corresponding to the anchor prediction result as the final output time interval.

[0328] Location difference is a measure of the degree of inconsistency between a time interval and the time position corresponding to the anchor point prediction result. Specifically, the anchor point prediction result contains information about the target object's time position, from which the corresponding time interval information can be extracted. By comparing the initial time interval or the repaired time interval with the time position corresponding to the anchor point prediction result, the greater the deviation, the greater the location difference; the smaller the deviation, the smaller the location difference. Furthermore, location difference can characterize the closeness between the time interval and the anchor point prediction result. The anchor point prediction result is an independent estimate unaffected by time coordinates. Using location difference as a metric, comparisons between two candidate time intervals can be completed without relying on annotation information such as the actual time interval.

[0329] During the processing, the positional differences between the initial time interval and the repair time interval and the anchor point prediction results can be obtained separately, serving as the basis for candidate selection. In practical applications, calculating the positional differences between the initial time interval and the repair time interval and the anchor point prediction results can be achieved through various methods.

[0330] One alternative method is to extract the corresponding time interval from the anchor point prediction results and calculate the positional difference based on the deviation of the two ends of the time interval from the corresponding endpoints of the corresponding interval. Specifically, obtain the start and end points of the interval corresponding to the anchor point prediction results; for the initial time interval, calculate the difference between its start point and the start point of the corresponding interval to obtain the start point difference, and calculate the difference between its end point and the end point of the corresponding interval to obtain the end point difference; for the repair time interval, obtain the corresponding start point difference and end point difference in the same way; for example, the two sets of differences can be combined according to the following formula to obtain the positional differences between the initial time interval and the repair time interval and the anchor point prediction results: Positional difference = Square of start point difference + Square of end point difference, where the start point difference is the difference between the start point of the time interval and the start point of the interval corresponding to the anchor point prediction results, and the end point difference is the difference between the end point of the time interval and the end point of the interval corresponding to the anchor point prediction results, and the start point difference and end point difference use the same time unit. This formula squares the deviations of the two endpoints and sums them to obtain the positional difference between the time interval and the anchor point prediction result. The squaring process can avoid the mutual cancellation when the deviations of the endpoints are in opposite directions, so that the positional difference stably reflects the overall degree of inconsistency between the time interval and the anchor point prediction result. The smaller the positional difference, the closer the time interval and the anchor point prediction result are.

[0331] Another alternative approach is to distribute the comparison between the time interval and the corresponding interval of the anchor point prediction result across multiple time locations within the interval, and then combine the comparison results from each time location to obtain the positional difference. Specifically, multiple time locations are selected within the time interval, and multiple corresponding time locations are selected within the interval corresponding to the anchor point prediction result. The difference between each group of corresponding time locations is calculated, and the squares of each difference are summed or averaged to obtain the positional difference between the time interval and the anchor point prediction result. Through multi-point comparison, the positional difference can reflect the overall deviation within the time interval.

[0332] Another alternative method is to decompose the positional difference into positional offset and length difference components, measure them separately, and then synthesize them. Specifically, the positional offset between the time interval and the corresponding interval of the anchor point prediction result is calculated; for example, the difference between the reference positions of the two intervals can be taken. Simultaneously, the difference in time length between the two intervals is calculated. The positional offset and length difference are then synthesized according to a predetermined method to obtain the positional difference between the time interval and the anchor point prediction result. By decomposing the measurement, the degree of deviation of the time interval relative to the anchor point prediction result in terms of both positioning location and coverage length can be characterized separately.

[0333] In addition, the above calculation methods can be combined, or an appropriate calculation method can be selected based on the shape of the time and location information parsed from the anchor point prediction results to obtain the usable location difference.

[0334] After obtaining two sets of positional differences, the two sets of positional differences can be compared to perform candidate selection. In practical applications, the time interval with the smallest positional difference is determined as the time interval corresponding to the target object, which can also be achieved through various methods.

[0335] One alternative approach is to compare the positional differences between the initial time interval and the repair time interval, identifying the candidate with the smaller positional difference as the target time interval. Specifically, if the positional difference between the repair time interval and the target time interval is smaller, the repair time interval is selected; similarly, if the positional difference between the initial time interval and the target time interval is smaller, the initial time interval is selected. By selecting the interval with the smallest positional difference based on the anchor point prediction results, a sample-by-sample selection of repair benefits is achieved.

[0336] Another alternative approach is to include the initial time interval and all repair time intervals in the selection when there are multiple repair time intervals. Specifically, these multiple repair time intervals could be those obtained by repairing with different interpolation intensities. The positional differences between the initial time interval and each repair time interval at the same anchor point are calculated, and all positional differences are compared. The candidate with the smallest positional difference is selected as the time interval corresponding to the target object. By uniformly comparing all candidates, the influence of differences in interpolation intensity selection on the selection of repair results can be avoided, and the time interval closest to the anchor point prediction result can still be selected even when multiple repair results coexist.

[0337] In addition, a validity check can be performed on the candidate time intervals before comparison. After removing candidates that cannot participate in the comparison normally, the positional differences among the remaining candidates can be compared to determine the time interval corresponding to the target object.

[0338] Furthermore, the comparison results of the two sets of location differences may be tied. In such cases, a strategy of preserving the baseline results can be adopted. In practical applications, when the location differences are the same, determining the initial time interval as the time interval corresponding to the target object can also be achieved through various methods.

[0339] One alternative approach is to directly retain the baseline result without intervention when two sets of location differences are the same or the comparison results are parallel, and determine the initial time interval as the time interval corresponding to the target object. Specifically, when two sets of location differences are the same, it indicates that the repair time interval and the initial time interval are not significantly different in terms of their similarity to the anchor point prediction results, making it impossible to determine whether the repair is better. In this case, the initial time interval is preferred to avoid introducing uncertain repair results, and in the case of parallel results, the determined processing can still output a definite result.

[0340] Another alternative approach is to first select within each of the multiple repair time intervals when there are multiple repair time intervals, and then compare them with the initial time interval. Specifically, compare the positional differences between each repair time interval and the anchor point prediction results, and determine the repair time interval with the smallest positional difference; then compare this repair time interval with the initial time interval. If the positional differences are the same, the initial time interval is determined as the time interval corresponding to the target object. By selecting within the intervals first and then comparing, the initial time interval is retained even when multiple repair time intervals are parallel to the baseline results, and the paralleling rule does not change with the number of candidates.

[0341] In addition, the parallel rules and the minimum selection can be implemented in a unified manner. For example, the candidate time intervals can be sorted from smallest to largest according to their positional differences. When the positional differences are the same, the initial time interval is sorted first, and the time interval at the top of the sorted interval is determined as the time interval corresponding to the target object.

[0342] Furthermore, situations may arise where anchor point prediction results cannot be parsed, meaning valid time interval information cannot be obtained. In such cases, anchor point references can be abandoned and a rollback performed. In practical applications, if at least one anchor point prediction result cannot be parsed, determining the initial time interval as the time interval corresponding to the target object can be achieved through various methods.

[0343] One alternative approach is to perform a parsing availability check on the anchor point prediction results before calculating the location difference. If parsing fails, a rollback is executed directly. Specifically, an attempt is made to parse the time interval information corresponding to the target object from the anchor point prediction results. If parsing is successful, the calculation of location differences and candidate selection continue based on the parsing results. If parsing fails, the anchor point prediction results are abandoned as a reference, and the initial time interval is directly determined as the time interval corresponding to the target object. By parsing first and then judging, the baseline results before intervention can be rolled back in time when the anchor point reference is missing, ensuring that the determination process still has usable output.

[0344] Another alternative approach is to perform a parsing availability check on each anchor point prediction result when there are multiple predictions. The parsable anchor point predictions are then used for further processing. Specifically, it is determined whether each anchor point prediction result can be parsed to obtain valid time interval information. Parsable anchor point predictions are used for calculating location differences and candidate selection. If all anchor point predictions cannot be parsed, the anchor point reference is abandoned, and the initial time interval is directly determined as the time interval corresponding to the target object. By checking the availability of each anchor point individually, the determination process can be completed using the remaining anchor points when some anchor points are unavailable.

[0345] In addition, the handling of parallel cases and unresolved cases can be unified, and the initial time interval can be retained when the anchor prediction results cannot be used to distinguish the superiority or inferiority of candidates.

[0346] In an alternative embodiment, the above determination process can also be made independent of labeled information and learned thresholds.

[0347] Specifically, the determination process does not use the actual time interval corresponding to the target object, nor does it use the learned threshold. It only relies on the anchor prediction result, a label-free reference, to calculate, compare, and select the position difference. Therefore, the above determination process can be directly used in unlabeled online scenarios.

[0348] In one optional embodiment, the anchor point prediction results can be multiple, such as prediction results obtained from multiple predictions, including prediction results in the form of direct audio transmission and prediction results in the form of rendering without superimposed time coordinates; the positional differences between the initial time interval and the repair time interval and each anchor point prediction result can be calculated separately to obtain multiple sets of positional differences, and comprehensive processing can be performed on the multiple sets of positional differences, such as summing or averaging the positional differences of each set; then, candidate selection can be performed based on the positional differences obtained from the comprehensive processing to determine the time interval corresponding to the target object.

[0349] In an optional embodiment, the method for determining candidate time intervals can be extended from selection to weighted fusion. Specifically, the weights corresponding to each candidate time interval can be determined according to the positional differences between each candidate time interval and the anchor point prediction results; the smaller the positional difference, the greater the corresponding weight. Then, the initial time interval and each repair time interval are weighted and fused according to their respective weights, and the fused time interval is determined as the time interval corresponding to the target object in the target multimodal data.

[0350] In the embodiments of this specification, by using the anchor point prediction results, which are unaffected by time coordinates, as a reference, the positional differences between the initial time interval and the repair time interval and the anchor point prediction results are calculated respectively. The time interval with the smallest positional difference is determined as the time interval corresponding to the target object, so that some effective repairs can be converted into stable benefits on a sample-by-sample basis. In cases where the positional differences are the same or the anchor point prediction results cannot be parsed, and it is impossible to confirm whether the repair is better, the initial time interval is determined as the time interval corresponding to the target object, the baseline results without intervention are retained, and the introduction of repairs without certainty is avoided. This ensures that a usable output time interval can be determined under all the input scenarios involved.

[0351] In one optional embodiment of this specification, a schematic diagram of a location request processing method is provided. Specifically, see... Figure 2 , Figure 2 A schematic diagram of the framework of a location request processing method provided according to an embodiment of this specification is shown.

[0352] like Figure 2 As shown, the processing framework may include: receiving target multimodal data at the input side, and a localization request for the target object in the target multimodal data; then, executing... Figure 2The visualization processing and temporal coordinate rendering in Module 1 are illustrated below. Module 1 uses visualization processing to obtain visual data, specifically including the conversion of audio data to a spectrogram and the processing of video data to obtain sampled frames through frame sampling. Temporal coordinate rendering is used to render the temporal coordinates onto the visual data, ensuring that the temporal coordinates and evidence content are located within the same visual data. This corresponds to printing timestamps from the video branch to the sampled frames and rendering time scales from the audio branch to the spectrogram time axis. Module 1 may also include rendering configuration screening to determine the rendering configuration based on the conditional space corresponding to position, density, and format. Module 1 performs deterministic processing on the model input side and does not involve parameter updates for the multimodal data processing model. The input obtained after processing in Module 1 is fed into the multimodal data processing model. This model has its parameters frozen and only performs forward inference, with the inference output being natural language text.

[0353] The inference process of the multimodal data processing model is connected to Module 2, which is used for temporal readout diagnosis. Specifically, it includes: training the temporal prediction probe layer by layer in each layer of the decoder to determine the readability of temporal information based on the prediction accuracy; and cross-modal transfer readout, comparing the transfer prediction results with the reference prediction results to determine the coordinate compatibility and thereby determine the target intervention layer. Module 2 is connected to Module 3, which is used for directional interpolation repair. Specifically, on the target intervention layer, the hidden state of the token corresponding to the time coordinate is directionally interpolated along the direction of the anchor hidden state according to the interpolation intensity. The anchor hidden state is the frame-aligned hidden state under the rendering without superimposed time coordinates. The calculation method of directional interpolation can be found in the description of directional interpolation in the previous embodiment, and will not be repeated here.

[0354] Furthermore, one output of the multimodal data processing model is connected to Module 4; the repair time interval obtained from Module 3 is also connected to Module 4. Module 4 is used for label-free conditional selection, including: label-free anchor points, which are anchor point prediction results obtained from rendering or direct audio transmission without superimposed time coordinates; candidates, including the initial time interval and repair time intervals corresponding to different interpolation intensities; and selection processing, which selects the candidate with the smallest positional difference from the label-free anchor point, retains the initial time interval when they are in a tie, and backs down to the initial time interval when the anchor point prediction result cannot be parsed. Here, the implementation example of positional difference is the minimum sum of the squared distances between the start and end endpoints.

[0355] Finally, Module 4 outputs the time interval corresponding to the target object in the target multimodal data. The accuracy of this output is evaluated according to the intersection-union ratio. Outputs that cannot be parsed are counted separately as invalid results.

[0356] In the embodiments of this specification, the processing framework shown in the figure is used to perform visualization processing and time coordinate rendering on the target multimodal data. The time coordinates are bound to the evidence content of the target to be located within the same visual data. This allows the trained multimodal data processing model to directly determine the time position corresponding to the evidence content based on the visible time coordinates in the input, completing time localization without adjusting model parameters. This can be applied to models with frozen parameters or those only providing an inference interface, broadening the application scope of time localization processing. Simultaneously, by performing temporal readout diagnosis on the model's inference process, the readability of time information in the latent state and its cross-modal coordinate compatibility can be distinguished, providing a mechanistic basis for determining the target intervention layer and giving the intervention a clear target. Based on this, rendering without superimposed time coordinates... Using the anchor point prediction results obtained through inference in the form of coloring or direct audio transmission as a reference, directional interpolation is performed on the latent state of the time coordinate corresponding to the word in the target intervention layer to obtain the repair time interval. Based on the positional difference, unlabeled conditional selection is performed between the initial time interval and the repair time interval. This can convert some effective repairs into stable localization benefits on a sample-by-sample basis. In the case of parallel situations or when the anchor point prediction results cannot be parsed, the initial time interval is retained to ensure that a usable time interval can be determined under various input situations. None of the above steps update the parameters of the multimodal data processing model, nor do they rely on the annotation information such as the real time interval corresponding to the target object. Overall, the accuracy and usability of multimodal data time localization are improved, and the feasibility of time localization processing in practical systems is enhanced.

[0357] Corresponding to the above method embodiments, this specification also provides embodiments of a location request processing system. Figure 3 A schematic diagram of the structure of a location request processing system according to one embodiment of this specification is shown. Figure 3 As shown, the location request processing system includes a front-end interface 302 and a location request processing module 304. The front-end interface 302 is used to receive target multimodal data and a positioning request for the target object in the target multimodal data, and send the target multimodal data and positioning request to the positioning request processing module 304. The location request processing module 304 is used to obtain target multimodal data and location requests, and execute the location request processing method as described above to obtain the time interval corresponding to the target object in the target multimodal data, and return it to the front-end interface 302.

[0358] The location request processing system executes the processing procedures in the location request processing method embodiments. It includes a front-end interface and a location request processing module. The front-end interface handles data exchange with the requester, while the location request processing module handles the core process of time-based location processing. Together, they form a complete processing chain from receiving target multimodal data and location requests to obtaining and returning the time interval corresponding to the target object in the target multimodal data. When deployed, the location request processing system can take the form of a software system, service, or middleware. In one optional embodiment, it independently provides time-based location services; in another optional embodiment, it is integrated into an existing media processing platform or inference service to supplement upper-layer applications with time-based location capabilities.

[0359] The front-end interface is the interface part of the location request processing system that provides access to the outside world. It receives target multimodal data and location requests for target objects within the multimodal data, sends the multimodal data and location requests to the location request processing module, and returns the processing result to the requesting party after processing. Specifically, the front-end interface can handle data interaction between the location request processing system and the user client or upstream system. One option is to interface with the user client or upstream system via API calls, obtaining the target multimodal data and location requests from the call requests, and returning a call response carrying a time range after processing. Another option is to interface with the upstream system via message transmission, obtaining the target multimodal data and location requests from task messages, and sending the time range as a return message after processing. Furthermore, the front-end interface can also have functions such as protocol adaptation and data verification. Protocol adaptation is used to convert data from different sources into a unified internal data format of the location request processing system, and data verification is used to check the format and integrity of the data during the receiving phase to ensure that the data entering the location request processing module meets the processing requirements.

[0360] The location request processing module is the core processing part of the location request processing system, executing the location request processing method. It acquires target multimodal data and the location request, executes various processes in the location request processing method embodiment, obtains the time interval corresponding to the target object in the target multimodal data, and returns the time interval to the front-end interface. Specifically, the processing performed by the location request processing module covers the complete processing chain in the location request processing method embodiment, including visualizing the target multimodal data to obtain visual data carrying evidence of the target object, rendering the time coordinates to the visual data to obtain multimodal data with time coordinates, performing inference with the multimodal data processing model input to the location request, and determining the time interval corresponding to the target object from the inference result. The location request processing module can also enable processing steps that depend on the internal state of the model in the location request processing method embodiment as needed; in this case, the multimodal data processing model needs to provide a corresponding internal state access interface.

[0361] The location request processing module can be implemented collaboratively by multiple internal components, such as a visualization processing unit, a rendering unit, and a model interaction unit. During processing, the visualization processing unit visualizes the target multimodal data to obtain visual data carrying the evidence content of the target object; the rendering unit renders the time coordinates to the visual data, placing the time coordinates and evidence content in the same visual data, forming multimodal data with time coordinates; the model interaction unit inputs the multimodal data with time coordinates and the location request into the multimodal data processing model, drives the model to perform inference based on the location request, and obtains the model's output results. When enabling processing steps that depend on the model's internal state, the model interaction unit can also read the hidden state generated during inference through the model's internal state access interface, and after the corresponding processing is completed, write the processed hidden state back into the inference process to drive the model to continue completing the inference.

[0362] During operation, the location request processing system receives the target multimodal data and location request submitted by the requester through the front-end interface. After the front-end interface completes protocol adaptation and data verification, it sends the target multimodal data and location request to the location request processing module. After obtaining the target multimodal data and location request, the location request processing module performs time positioning according to the processing procedure in the location request processing method embodiment, and returns the obtained time interval to the front-end interface, which provides it to the requester.

[0363] When necessary, the location request processing module can also enable further auxiliary processing steps such as diagnosis, intervention, and selection during or after the execution of time-based location. For example, it can perform diagnostic analysis on the internal states related to the time coordinates during model inference, intervene in intermediate states that cause location deviations, and select and determine among candidate location results. Whether each auxiliary processing step is enabled can be determined based on the system's deployment configuration and the access capabilities provided by the multimodal data processing model.

[0364] The collaboration between the front-end interface and the location request processing module can be presented to the user through an interactive interface. For example, the user can upload target multimodal data containing the target object and input a location request for the target object through the interactive interface. After receiving the target multimodal data and location request submitted by the interactive interface, the front-end interface sends both to the location request processing module. After completing the time location processing, the location request processing module returns the time interval corresponding to the target object in the target multimodal data to the front-end interface, which is then displayed to the user through the interactive interface. Based on the displayed time interval, the user can perform downstream operations such as playback location, clip editing, or content review on the target multimodal data. When the location request processing system is embedded in the upstream task flow, the upstream system only needs to submit the target multimodal data and location request through the front-end interface to obtain the time interval corresponding to the target object, which can then be directly used for subsequent automated processing.

[0365] In an optional embodiment, the deployment mode of the location request processing system can be divided into different levels according to the access permissions to the multimodal data processing model. The difference between each level is only whether the processing link that depends on the internal state of the model is enabled, without involving the adjustment of model parameters. The location request processing system can select the matching level for deployment according to the access capabilities provided by the model in the actual operating environment.

[0366] For example, in the tier that only performs rendering and basic time positioning processing, all processing can be completed by simply calling the inference interface of the multimodal data processing model during operation. Visualization and time coordinate rendering are performed outside the model. Therefore, the positioning request processing system in this tier can be used as a capability enhancement middleware, mounted on any already deployed model, and can complete time positioning processing normally even if the model only provides an inference interface.

[0367] When processing steps that rely on the model's internal state need to be enabled, the corresponding tier also requires the multimodal data processing model to provide an interface for reading the internal state in order to read the hidden states generated during inference. Tiers with full processing capabilities can also write back the processed hidden states to the inference process after reading them, in order to support the enabling of intervention-type processing steps. The model parameters remain unchanged during the operation of each tier, without altering the model's weights or structure. Deploying or switching tiers does not require the model provider to perform training or fine-tuning.

[0368] In this embodiment, the various processes in the location request processing method are implemented through a modular composition of a front-end interface and a location request processing module. This allows the time location processing capability to be provided externally in the form of software systems, services, or middleware, lowering the threshold for practical deployment of the location request processing method. Through the division of labor and cooperation between the front-end interface and the location request processing module in the receiving, forwarding, processing, and return stages, the location request processing system can seamlessly interface with user terminals and upstream systems, facilitating the embedding of time location capabilities into existing media processing and content review processes. Furthermore, by supporting hierarchical deployment based on model access permissions, operating environments with different access permissions can obtain matching time location capabilities while maintaining unchanged model parameters, expanding the application scope of the location request processing method embodiment.

[0369] The above is an illustrative scheme of a location request processing system according to this embodiment. It should be noted that the technical solution of this location request processing system and the technical solution of the location request processing method described above belong to the same concept. For details not described in detail in the technical solution of the location request processing system, please refer to the description of the technical solution of the location request processing method described above.

[0370] Figure 4 A structural block diagram of a computing device 400 according to one embodiment of this specification is shown. The components of the computing device 400 include, but are not limited to, a memory 410 and a processor 420. The processor 420 is connected to the memory 410 via a bus 430, and a database 450 is used to store data.

[0371] The computing device 400 also includes an access device 440, which enables the computing device 400 to communicate via one or more networks 460. Examples of these networks include a Public Switched Telephone Network (PSTN), a Local Area Network (LAN), a Wide Area Network (WAN), a Personal Area Network (PAN), or a combination of communication networks such as the Internet. The access device 440 may include one or more of any type of wired or wireless network interface (e.g., a Network Interface Controller (NIC)), such as an IEEE 802.11 Wireless Local Area Network (WLAN) interface, a Worldwide Interoperability for Microwave Access (WiMAX) interface, an Ethernet interface, a Universal Serial Bus (USB) interface, a cellular network interface, a Bluetooth interface, or a Near Field Communication (NFC) interface.

[0372] In one embodiment of this specification, the aforementioned components of the computing device 400 and Figure 4 Other components, not shown, can also be connected to each other, for example, via a bus. It should be understood that... Figure 4 The block diagram of the computing device shown is for illustrative purposes only and is not intended to limit the scope of this specification. Those skilled in the art can add or replace other components as needed.

[0373] The computing device 400 can be any type of stationary or mobile computing device, including mobile computers or mobile computing devices (e.g., tablet computers, personal digital assistants, laptop computers, notebook computers, netbooks, etc.), mobile phones (e.g., smartphones), wearable computing devices (e.g., smartwatches, smart glasses, etc.) or other types of mobile devices, or stationary computing devices such as desktop computers or personal computers (PCs). The computing device 400 can also be a mobile or stationary server.

[0374] The processor 420 is configured to execute the following computer-executable instructions, which, when executed by the processor, implement the steps of the above-described location request processing method.

[0375] The above is an illustrative scheme of a computing device according to this embodiment. It should be noted that the technical solution of this computing device and the technical solution of the location request processing method described above belong to the same concept. For details not described in detail in the technical solution of the computing device, please refer to the description of the technical solution of the location request processing method described above.

[0376] An embodiment of this specification also provides a computer-readable storage medium storing computer-executable instructions that, when executed by a processor, implement the steps of the location request processing method described above.

[0377] The above is an illustrative scheme of a computer-readable storage medium according to this embodiment. It should be noted that the technical solution of this storage medium and the technical solution of the location request processing method described above belong to the same concept. For details not described in detail in the technical solution of the storage medium, please refer to the description of the technical solution of the location request processing method described above.

[0378] An embodiment of this specification also provides a computer program product, including computer-executable instructions that, when executed by a processor, implement the steps of the location request processing method described above.

[0379] The above is an illustrative scheme of a computer program product according to this embodiment. It should be noted that the technical solution of this computer program product and the technical solution of the above-described location request processing method belong to the same concept. For details not described in detail in the technical solution of the computer program product, please refer to the description of the technical solution of the above-described location request processing method.

[0380] The foregoing has described specific embodiments of this specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims may be performed in a different order than that shown in the embodiments and may still achieve the desired result. Furthermore, the processes depicted in the drawings do not necessarily require the specific or sequential order shown to achieve the desired result. In some embodiments, multitasking and parallel processing are possible or may be advantageous.

[0381] Computer-executable instructions include computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. Computer-readable storage media can include: any entity or device capable of carrying computer program code, recording media, USB flash drives, portable hard drives, magnetic disks, optical disks, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc. It should be noted that the content included in computer-readable media can be appropriately added or removed according to the requirements of patent practice. For example, in some regions, according to patent practice, computer-readable media do not include electrical carrier signals and telecommunication signals.

[0382] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that the embodiments in this specification are not limited to the described order of actions, because according to the embodiments in this specification, some steps can be performed in other orders or simultaneously. Furthermore, those skilled in the art should also understand that the embodiments described in this specification are all preferred embodiments, and the actions and modules involved are not necessarily essential to the embodiments in this specification.

[0383] In the above embodiments, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions in other embodiments.

[0384] The preferred embodiments disclosed above are merely illustrative of this specification. Optional embodiments do not exhaustively describe all details, nor do they limit the invention to the specific implementations described. Clearly, many modifications and variations can be made based on the embodiments described herein. These embodiments are selected and specifically described in this specification to better explain the principles and practical applications of the embodiments, thereby enabling those skilled in the art to better understand and utilize this specification. This specification is limited only by the claims and their full scope and equivalents.

Claims

1. A location request processing method, comprising: Acquire target multimodal data, and request the location of target objects within the target multimodal data; The target multimodal data is visualized to obtain at least one visual data, wherein the visual data carries the evidentiary content of the target object; The time coordinates are rendered onto the visual data to obtain multimodal data with time coordinates, wherein the time coordinates and the evidence content are located in the same visual data; The time-coordinated multimodal data and the positioning request are input into a preset multimodal data processing model. Based on the positioning request, the time-coordinated multimodal data is inferred to obtain the time interval corresponding to the target object in the target multimodal data.

2. The method according to claim 1, wherein the target multimodal data includes audio data, and the visual data includes a spectrogram; The visualization processing of the target multimodal data to obtain at least one visual data includes: The audio data is subjected to spectral conversion to obtain the spectrogram; The step of rendering the time coordinates onto the visual data to obtain multimodal data with time coordinates includes: The time scale of the audio data is used as the time coordinate to render the spectrogram, resulting in a spectrogram with a time axis.

3. The method according to claim 1, wherein the target multimodal data includes video data, and the visual data includes sampled frames; The visualization processing of the target multimodal data to obtain at least one visual data includes: The video data is sampled frame by frame to obtain the sampled frame; The step of rendering the time coordinates onto the visual data to obtain multimodal data with time coordinates includes: The timestamp of the video data is printed as the time coordinate to the sampling frame to obtain a sampling frame with a timestamp.

4. The method according to claim 3, further comprising, before printing the timestamp of the video data as the time coordinate to the sampling frame: Based on the timestamp attribute parameters of the video data, a condition space for candidate rendering configurations is constructed. For each candidate rendering configuration in the condition space, the sampled frames with the timestamps of the video data printed according to each candidate rendering configuration are inferred with the positioning request input into the multimodal data processing model to obtain the positioning evaluation result corresponding to each candidate rendering configuration; Based on the positioning evaluation results, the target rendering configuration is determined from the condition space; The step of printing the timestamp of the video data as the time coordinate to the sampling frame to obtain a timestamped sampling frame includes: According to the target rendering configuration, the timestamp of the video data is printed to the sampling frame as the time coordinate to obtain the sampling frame with timestamp.

5. The method according to claim 1, further comprising, before inputting the time-coordinated multimodal data and the positioning request into a preset multimodal data processing model: Freeze the parameters of the multimodal data processing model; The step of reasoning about the time-coordinated multimodal data based on the positioning request to obtain the time interval corresponding to the target object in the target multimodal data includes: Using the multimodal data processing model with its parameters frozen, forward inference is performed on the multimodal data with time coordinates based on the positioning request to obtain the time interval corresponding to the target object in the target multimodal data, wherein the parameters of the multimodal data processing model remain unchanged during the forward inference process.

6. The method according to any one of claims 1 to 5, wherein the step of inferring from the time-coordinated multimodal data based on the positioning request to obtain the time interval corresponding to the target object in the target multimodal data comprises: Based on the positioning request, inference is performed on the multimodal data with time coordinates to obtain the initial time interval of the target object; Based on the initial time interval, the time interval corresponding to the target object in the target multimodal data is determined.

7. The method according to claim 6, further comprising, before determining the time interval corresponding to the target object in the target multimodal data based on the initial time interval: The initial time interval is validated based on preset validation rules. Determining the time interval corresponding to the target object in the target multimodal data based on the initial time interval includes: If the legality verification passes, the time interval corresponding to the target object in the target multimodal data is determined based on the initial time interval.

8. The method according to claim 6, wherein the step of inferring the multimodal data with time coordinates based on the positioning request to obtain the initial time interval of the target object includes: Based on the location request, inference is performed on the multimodal data with time coordinates to obtain text inference results; The initial time interval of the target object is obtained by parsing the text reasoning results.

9. The method according to claim 6, after inferring the multimodal data with time coordinates based on the positioning request to obtain the initial time interval of the target object, further comprising: Obtain the hidden states of each word generated during the inference process of the multimodal data processing model; Determine the readability of the time information of the target object in the hidden state, and the coordinate compatibility of the time information among the various modalities of the target multimodal data; The target intervention layer is determined based on the readability and coordinate compatibility. Determining the time interval corresponding to the target object in the target multimodal data based on the initial time interval includes: Based on the target intervention layer and the initial time interval, the time interval corresponding to the target object in the target multimodal data is determined.

10. The method according to claim 9, wherein determining the readability of the time information of the target object in the latent state, and the coordinate compatibility of the time information among the modalities of the target multimodal data, comprises: Based on the hidden state of the lexical units corresponding to the time coordinates, time prediction probes are trained on each layer of the decoder of the multimodal data processing model. Based on the prediction accuracy of the time information by the time prediction probe at each layer, the readability of the time information of the target object in the hidden state is determined. The time prediction probe is applied to the hidden state of a word in a different modality than the word in the time coordinate to obtain the migration prediction result. Based on the migration prediction results and the preset reference prediction results, the degree of coordinate compatibility of the time information among the various modes of the target multimodal data is determined.

11. The method according to claim 9, wherein determining the time interval corresponding to the target object in the target multimodal data based on the target intervention layer and the initial time interval comprises: Obtain the anchor point prediction result and the anchor point hidden state corresponding to the anchor point prediction result, wherein the anchor point prediction result is the prediction result obtained by inference of the target multimodal data without superimposing the time coordinate, and the anchor point hidden state is the hidden state of the word corresponding to the time coordinate aligned frame by frame during the process of inferring the anchor point prediction result. On the target intervention layer, the hidden state of the word corresponding to the time coordinate is oriented and interpolated along the direction of the hidden state of the anchor point, and reasoning is performed based on the interpolated hidden state to obtain the repair time interval; Based on the repair time interval and the initial time interval, the time interval corresponding to the target object in the target multimodal data is determined.

12. The method according to claim 11, wherein the step of directional interpolating the hidden state of the lexical corresponding to the time coordinate along the direction of the anchor point hidden state includes: Based on the deviation direction and deviation magnitude between the initial time interval and the anchor point prediction result, the target interpolation intensity of the directional interpolation is determined; According to the target interpolation intensity, the hidden state of the lexical corresponding to the time coordinate is oriented and interpolated along the direction of the anchor point hidden state.

13. The method according to claim 11, wherein determining the time interval corresponding to the target object in the target multimodal data based on the repair time interval and the initial time interval comprises: Calculate the positional differences between the initial time interval and the repair time interval and at least one of the anchor point prediction results, respectively; The time interval with the smallest positional difference is determined as the time interval corresponding to the target object in the target multimodal data; When the positional differences are the same, the initial time interval is determined as the time interval corresponding to the target object in the target multimodal data; If at least one of the anchor point prediction results cannot be resolved, the initial time interval is determined as the time interval corresponding to the target object in the target multimodal data.

14. A location request processing system, comprising a front-end interface and a location request processing module; The front-end interface is used to receive target multimodal data and a location request for a target object in the target multimodal data, and to send the target multimodal data and the location request to the location request processing module. The location request processing module is used to acquire the target multimodal data and the location request, and execute the location request processing method as described in any one of claims 1 to 13 to obtain the time interval corresponding to the target object in the target multimodal data, and return it to the front-end interface.

15. A computing device, comprising: Memory and processor; The memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions to implement the steps of the location request processing method according to any one of claims 1 to 13.

16. A computer-readable storage medium storing computer-executable instructions that, when executed by a processor, implement the steps of the location request processing method according to any one of claims 1 to 13.

17. A computer program product comprising computer-executable instructions that, when executed by a processor, implement the steps of the location request processing method according to any one of claims 1 to 13.

Images