Video-based query method and electronic device

Abstract

This disclosure presents a video-based query method and electronic device, relating to the field of computer vision technology. The electronic device includes a processor and a memory storing computer program instructions. The processor executes the computer program instructions to: receive a query request for an original video; decompose the query request to obtain a task sequence, the task sequence including multiple tasks to be executed sequentially; according to the task order of the multiple tasks in the task sequence, sequentially call the target task execution module corresponding to each task to process the original video and obtain the processing result of each task; and determine a first query result corresponding to the query request based on the processing result of each task. This processor can reduce the consumption of image frames, reduce computational power consumption, and improve the accuracy of the query results.

Description

Technical Field

[0001] This disclosure relates to the field of computer vision technology, and in particular to a video-based query method, electronic device, computer-readable storage medium, and computer program product. Background Technology

[0002] With the rapid development of video capture equipment and network transmission technology, video data is experiencing explosive growth, and users' needs for understanding and retrieving video content are becoming increasingly urgent. Currently, to meet users' query needs for long videos, keyframe extraction is commonly used to generate query results. This involves uniformly sampling the video to extract keyframes and then generating query results based on these keyframes.

[0003] However, long videos differ from short videos. Long videos have significant data volume and temporal structure characteristics. They contain a massive number of video frames (for example, a 20-minute video contains tens of thousands of frames) and complex spatiotemporal changes. Simply relying on discrete key video frames to generate query results requires searching based on a large number of video frames, which increases computing power consumption and also leads to lower accuracy of query results. Summary of the Invention

[0004] This disclosure provides a video-based query method, electronic device, computer-readable storage medium, and computer program product that can reduce the retrieval consumption of multiple video frames, further reduce computing power consumption, and improve the accuracy of query results.

[0005] In a first aspect, embodiments of this disclosure provide an electronic device, including: a processor and a memory storing computer program instructions; The processor executes the computer program instructions to achieve: Receive query requests for the original video; The query request is decomposed to obtain a task sequence, which includes multiple tasks to be executed sequentially; According to the task order of multiple tasks in the task sequence, the target task execution module corresponding to each task is called in turn to process the original video and obtain the processing result of each task. Based on the processing results of each of the tasks, a first query result corresponding to the query request is determined.

[0006] Secondly, embodiments of this disclosure provide a video-based query method, including: Receive query requests for the original video; The query request is decomposed to obtain a task sequence, which includes multiple tasks to be executed sequentially; According to the task order of multiple tasks in the task sequence, the target task execution module corresponding to each task is called in turn to process the original video and obtain the processing result of each task. Based on the processing results of each of the tasks, a first query result corresponding to the query request is determined.

[0007] Thirdly, embodiments of this disclosure provide a computer-readable storage medium storing computer program instructions that, when executed by a processor, implement the video-based query method as described in the second aspect.

[0008] Fourthly, embodiments of this disclosure provide a computer program product, including a computer program that, when executed by a processor, implements the video-based query method as described in the second aspect.

[0009] The electronic device of this disclosure receives a query request for the original video through a processor. Before generating the query result based on the original video, a decomposition step is added to the query request. That is, the received query request is decomposed into multiple tasks to be executed sequentially, constructing an ordered and executable task sequence. By decomposing the complex query request in advance, the information required for each step of the task execution can be accurately located, reducing the process of indiscriminate global analysis of the original video and reducing the calculation of invalid frames from the source, thereby effectively reducing the overall consumption of computing resources. Furthermore, after obtaining the task sequence corresponding to the query request, the task execution modules adapted to each task are called in sequence according to the task sequence to perform unidirectional and multi-dimensional targeted analysis and processing on the original video, and obtain the independent processing results corresponding to each task. It is not necessary to traverse all redundant frames of the video, nor is it necessary to repeatedly calculate the global spatiotemporal features of the video. At the same time, relying on the task decomposition and multi-dimensional targeted video processing mode, the analysis requirements of each decomposed task can be matched, avoiding the defects of coarse sampling, loss of key details caused by global blind analysis, and mismatch of spatiotemporal information in the understanding of long videos in related technologies. Finally, the electronic device determines the query result corresponding to the query request based on the processing results of each task, which not only reduces the consumption of computing resources, but also improves the accuracy of the query result. Attached Figure Description

[0010] Figure 1 This is a schematic diagram of the structure of an electronic device provided in one embodiment of the present disclosure; Figure 2 This is one of the flowcharts illustrating how a processor executes instructions to implement a video-based query method according to an embodiment of this disclosure; Figure 3This is a second schematic flowchart illustrating how a processor executes instructions to implement a video-based query method according to an embodiment of this disclosure. Figure 4 This is the third flowchart illustrating a processor executing instructions to implement a video-based query method according to an embodiment of this disclosure; Figure 5 This is the fourth flowchart illustrating a processor executing instructions to implement a video-based query method according to an embodiment of this disclosure; Figure 6 This is the fifth flowchart illustrating a processor executing instructions to implement a video-based query method according to an embodiment of this disclosure; Figure 7 This is a sixth flowchart illustrating the processor executing instructions to implement a video-based query method according to an embodiment of this disclosure; Figure 8 This is the seventh flowchart illustrating the processor executing instructions to implement a video-based query method according to an embodiment of this disclosure; Figure 9 This is the eighth flowchart illustrating a processor executing instructions to implement a video-based query method according to an embodiment of this disclosure. Detailed Implementation

[0011] The features and exemplary embodiments of various aspects of this disclosure will now be described in detail. To make the objectives, technical solutions, and advantages of this disclosure clearer, the following detailed description, in conjunction with the accompanying drawings and specific embodiments, will provide a further detailed description. It should be understood that the specific embodiments described herein are intended only to explain this disclosure and not to limit it. For those skilled in the art, this disclosure can be implemented without some of these specific details. The following description of the embodiments is merely to provide a better understanding of this disclosure by illustrating examples.

[0012] It should be noted that, in this document, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising..." does not exclude the presence of additional identical elements in the process, method, article, or apparatus that includes the element.

[0013] Application Overview With the rapid development of video capture equipment and network transmission technology, video data is experiencing explosive growth, and users' needs for understanding and retrieving video content are becoming increasingly urgent. Currently, to meet users' query needs for long videos, a common approach is to perform global uniform frame extraction on the input long video, selecting discrete frames as all representative materials of the video, and generating query results based solely on these discrete frames.

[0014] However, long videos differ from short videos. Long videos have significant data volume and temporal structure characteristics, containing massive amounts of video frames (e.g., a 20-minute video contains tens of thousands of frames) and complex spatiotemporal changes. Uniform sampling rules do not distinguish between valid semantic information within frames, indiscriminately extracting static frames without visual changes, event progression, or target interaction, thus affecting the accuracy of query results. Moreover, a large number of discrete frames carrying repetitive and invalid information participate in the generation of query results simultaneously, significantly increasing the computational load on processors and video memory, resulting in substantial consumption of computing resources. While reducing the number of discrete frames extracted can reduce computational consumption, it will result in the loss of a large amount of temporal details; conversely, increasing the number of discrete frames sampled to retain video information will cause a surge in the total number of frames processed, inevitably leading to a significant increase in computational overhead. Therefore, relying solely on discrete frames to generate query results requires calculations on a large number of discrete frames, increasing computational consumption and resulting in lower accuracy of query results.

[0015] To address the aforementioned technical problems, this disclosure provides a video-based query method and electronic device. The electronic device includes a processor and a memory storing computer program instructions. When the processor executes the computer program instructions, it performs the following: receiving a query request for the original video; decomposing the query request to obtain a task sequence, the task sequence including multiple tasks executed sequentially; sequentially calling the target task execution module corresponding to each task according to the task order in the task sequence to process the original video and obtain the processing results of each task; and determining the first query result corresponding to the query request based on the processing results of each task. This reduces the retrieval consumption of multiple video frames, further reduces the consumption of computing resources, and improves the accuracy of the query results.

[0016] Exemplary System In view of the above-mentioned technical problems, this disclosure provides a video-based query system, which includes an electronic device and a video acquisition device. The electronic device may include a processor and a memory storing computer program instructions.

[0017] For example, the electronic device can be an edge device in any field. It can be a device with data processing capabilities, including but not limited to servers, cloud servers, edge computing devices, personal computers, workstations, laptops, smartphones, tablets, smart terminals, in-vehicle terminals, robots, smart cockpit devices, smart security devices, and other electronic devices with video processing capabilities. The video acquisition device can be a camera, webcam, or other video acquisition device. In autonomous driving applications, the video acquisition device can be a dashcam.

[0018] In practice, the video acquisition device acquires the original video and sends it to the electronic device. The electronic device receives a query request for the original video, decomposes the query request to obtain a task sequence, which includes multiple tasks to be executed sequentially. According to the task order of the multiple tasks in the task sequence, the target task execution module corresponding to each task is called in turn to process the original video and obtain the processing results of each task. Based on the processing results of each task, the first query result corresponding to the query request is determined.

[0019] Based on the aforementioned video-based query system, the electronic device provided in this disclosure can be deployed in vehicle terminals, robots, intelligent camera devices, intelligent security devices, or other devices with video processing capabilities. The electronic device can understand the collected video and respond to user-inputted query requests, retrieving video segments, targets, events, behaviors, or semantic information from the video that meet the query conditions.

[0020] For example, in a vehicle scenario, users can query whether a specific target, traffic event, or driving behavior has occurred during the vehicle's operation; in a robot scenario, users can query whether a specified object, person, action, or event exists in historical videos, and the corresponding video clip or query result will be returned.

[0021] It should be noted that the above query request may include any one or two of the following: target, person, scenario, time, event, behavior, spatial location, trajectory, and semantic relationship. The query request can be determined based on the query information entered by the user in a specific application scenario.

[0022] Exemplary electronic devices Figure 1 This is a schematic diagram of the structure of an electronic device provided in one embodiment of the present disclosure. Figure 2 This is one of the flowcharts illustrating how a processor implements a video-based query method in one embodiment of this disclosure. For example... Figure 1 As shown, the electronic device 100 includes: a processor 110 and a memory 120 storing computer program instructions; the processor 110 executes the computer program instructions to implement, for example... Figure 2 The video-based query method is shown.

[0023] In some embodiments, the electronic device may be an edge device, which may include, but is not limited to, mobile terminals and vehicle-mounted terminals. The computer program instructions may be applied to any operating system.

[0024] like Figure 1 As shown, the processor 110 can be a general-purpose central processing unit (CPU), a graphics processing unit (GPU), or a neural processing unit (NPU) to accelerate the tensor computation and feature extraction process of large language models.

[0025] Memory 120 may include mass storage for data or instructions. For example, and not limitingly, memory 120 may include a hard disk drive (HDD), floppy disk drive, flash memory, optical disk, magneto-optical disk, magnetic tape, or Universal Serial Bus (USB) drive, or a combination of two or more of these. Where appropriate, memory 120 may include removable or non-removable (or fixed) media. In a particular embodiment, memory 120 is a non-volatile solid-state memory.

[0026] Memory may include read-only memory (ROM), random access memory (RAM), disk storage media devices, optical storage media devices, flash memory devices, and electrical, optical, or other physical / tangible memory storage devices. Therefore, typically, memory includes one or more tangible (non-transitory) computer-readable storage media (e.g., memory devices) encoded with software including computer-executable instructions, and when the software is executed (e.g., by one or more processors), it is operable to perform the operations described with reference to the method according to one aspect of this disclosure.

[0027] In an alternative embodiment, the electronic device may also include input and output devices, which are interconnected via a bus system and / or other forms of connection mechanism.

[0028] The input device may also include, for example, a keyboard, a mouse, etc.

[0029] The output device can output various information to the outside, including, for example, a display, a speaker, a printer, and a communication network and its connected remote output devices.

[0030] The processor 110 determines the query result by reading and executing computer program instructions stored in the memory 120, such as... Figure 2 As shown, the processor executes computer program instructions to implement S210 to S240.

[0031] S210, Receive a query request for the original video.

[0032] In some embodiments, the electronic device may store the original video in its memory before receiving a query request for the original video from a user or robot. Alternatively, in other embodiments, the original video is not stored in the electronic device's memory but is stored in the cloud, and the electronic device retrieves the original video from the cloud via an interface based on the received query request. Or, in still other embodiments, the electronic device may receive the original video input by the user simultaneously with receiving the query request for the original video.

[0033] In this embodiment of the disclosure, the original video corresponding to the query request can be one or more. For example, in an autonomous driving scenario, the vehicle terminal can collect videos from multiple perspectives through cameras with multiple perspectives, and these videos from multiple perspectives can be used as the original video. When there are multiple original videos, based on the user's query request, any one of the multiple videos can be used as the original video, or multiple videos can be used together as the original video.

[0034] In some embodiments, in autonomous driving applications, the original video can be in-vehicle monitoring video captured by a dashcam. In robotics applications, the original video can be recorded video captured by a camera on an industrial inspection robot. In smart home applications, the original video can be monitoring video captured by an AI camera. The original video can also be a documentary or drone aerial footage.

[0035] In this embodiment of the disclosure, the query request is for the content in the original video, and the query request can be text, voice or video input by the user or robot.

[0036] In some embodiments, when the query request is in the form of voice, the electronic device may transcribe the voice information to obtain a query request in text form.

[0037] In some embodiments, when the query request is a video, the electronic device can extract audio information from the video information. The audio information is then transcribed to obtain the query request in text form.

[0038] In this embodiment of the disclosure, the query request may include a query object or a query target, which may serve as an object query element. Optionally, the query request may also include time information, i.e., a time query element. The query object or query target may be a person, scene, behavior, event, trajectory, or state, etc.

[0039] For example, in an autonomous driving scenario, if a user's query is "find recently observed traffic lights," the query object is the traffic lights, and the time information is "recently observed." If the user's query is "did any pedestrians cross the road in the last minute?", the query object is the pedestrians crossing the road, and the time information is "last minute." If the user's query is "please query the trajectory of the white vehicle that just passed by," the query object is the trajectory of the white vehicle, and the time information is "just passed by." In the field of robotics, if a user's query is "was any equipment alarmed at 11:00 AM?", the query object is the alarmed equipment, and the time information is "11:00 AM." Or, if a user's query is "did anyone enter the lab today?", the query object is people, and the time information is "today."

[0040] In another example, if the electronic device simultaneously receives both the original video input by the user and a query request, and the query request may also include identification information for the original video, then the identification information is used to identify the video specifically referred to in the query request. For example, if the user inputs an indoor surveillance video of a furniture scene, designated as video 'a', the query request could include information about the query target for video 'a'. For instance, the query request could be "How is the bottle cap placed at the beginning of video 'a'?"

[0041] S220. Decompose the query request to obtain a task sequence, which includes multiple tasks to be executed sequentially.

[0042] In some embodiments, the query request corresponding to a Long Video Understanding (LVU) query typically contains rich query elements. To efficiently find the content corresponding to the query request from the original video, a comprehensive understanding of the query request is necessary. The query request is decomposed so that the query process of the original video can be divided into multiple tasks at different stages based on different query elements. Appropriate task execution modules are invoked for different tasks to extract key information corresponding to each task from the original video, thereby quickly determining the query results corresponding to the query request.

[0043] In some embodiments, the query request can be decomposed into multiple query elements, such as time query elements and object query elements. The time query elements can then be considered as one task, and the object query elements as another task, thus forming a task sequence by combining the time query elements and object query elements in that order. Alternatively, if the query elements include time query elements, object query elements, and query status, then the time query elements can be considered as one task, the object query elements as another task, and the query status as a third task, thus forming a task sequence by combining the time query elements, object query elements, and query status in that order.

[0044] The dependencies between tasks can be logically ordered based on each query element to reduce the number of video frames searched when querying long videos. Alternatively, electronic devices can pre-set the dependencies between tasks.

[0045] In other embodiments, the query request can be decomposed based on the constraints of the task execution modules, breaking down complex query requests into a sequence of tasks that are executed sequentially. The constraints of the task execution modules include their functions and dependencies. Specifically, the functions of multiple task execution modules and the dependencies between each task execution model are pre-defined. Each task execution module is a model, computational operator, or computational module capable of implementing any query function. For example, task execution modules may include modules for time-based queries, video editing, object-based queries, and object status queries. Each task execution module can implement its corresponding function based on an existing large language model.

[0046] In another example, the query request is "How are the bottle caps placed at the beginning of the video?". Based on the functions and dependencies of the task execution modules, the query request is decomposed into multiple tasks corresponding to "At the beginning of the video", "bottle caps", and "How are these bottle caps placed", as well as the order of these tasks. "At the beginning of the video" corresponds to the time determination task execution module, "bottle caps" corresponds to the retrieval task execution module, and "How are these bottle caps placed" corresponds to the identification task execution module.

[0047] In some embodiments, the sentence components of the query request can be decomposed to obtain keywords with different parts of speech, and the query request can be decomposed according to the keywords to obtain subquery requests. Based on the decomposed subquery requests, a task sequence including multiple tasks to be executed sequentially is generated.

[0048] It should be noted that each task corresponds to at least one task execution module. A task execution module can be a model, a computation operator, or a computation module. For example, the task execution module could be the ASR_Search computation module.

[0049] S230. According to the task order of multiple tasks in the task sequence, call the target task execution module corresponding to each task in turn to process the original video and obtain the processing results of each task.

[0050] The target task execution module can be a model, a computation operator, or a computation module. For example, the target task execution module could be the ASR_Search computation module.

[0051] In some embodiments, an execution engine is used to sequentially call the target task execution modules corresponding to each task according to the task order of multiple tasks in a task sequence. These target task execution modules process the original video to obtain the processing results for each task. For example, the task execution modules include a task execution module capable of time querying, a task execution module capable of video editing, a task execution module capable of object querying, and a task execution module capable of object status querying. The task order includes a task sequence composed of the order of time query elements, object query elements, and query status. Therefore, according to the task order of multiple tasks in the task sequence, the target task execution modules corresponding to each task are called sequentially, including: a task based on time query elements calling the task execution module capable of time querying to query the video; a task based on object query elements calling the task execution module capable of object querying to query the video or the processing results of the previous stage; and a task based on query status calling the task execution module capable of object status querying to query the video and / or the processing results of the previous stage to obtain the final query result.

[0052] It should be noted that the output of the target task execution module corresponding to the previous task can be used as input data for the target task execution module corresponding to the next task. For example, the task execution module that performs time queries can provide the time information obtained from the video query as input to the video editing task execution module, enabling the video editing task execution module to edit the video based on the time information. Alternatively, the object recognition task execution module can identify objects in the video and send video segments or frames containing those objects to the state recognition task execution module. The state recognition task execution module then identifies the video segments or frames to obtain the final state recognition result. Each target task execution module can output output results with different dimensions of information, such as time information, video segments, and visual frames. For example, the time query task execution module outputs time information after querying a video, and the object recognition task execution module outputs video segments or frames containing those objects after identifying them in the video.

[0053] In some embodiments, each target task execution module is deployed on an electronic device. The processor executes computer program instructions to invoke and run each target task execution module.

[0054] In some embodiments, each target task execution module is deployed on a cloud server. The processor executes computer program instructions to invoke and run the target task execution modules on the cloud server.

[0055] S240. Based on the processing results of each task, determine the first query result corresponding to the query request.

[0056] In some embodiments, the preprocessing has decomposed the user’s complex query request into multiple tasks to be executed sequentially, and performed unidirectional multi-dimensional directional processing on the original long video. Each task will output a corresponding processing result. Different processing results correspond to independent dimensional information such as time information, video segments, and visual frames. Based on the processing results of each task, the first query result corresponding to the query request is determined. The obtained first query result is more accurate and better meets the user’s needs.

[0057] In some embodiments, the processing results of each task are merged to obtain a fused result. The fused result is determined as the first query result corresponding to the query request.

[0058] In some embodiments, the task execution module corresponding to the last task is determined according to the task order. The processing result output by the last task execution module is determined as the first query result corresponding to the query request.

[0059] In some embodiments, after determining the first query result corresponding to the query request, the first query result is output. Optionally, the processing results output by all task execution modules may also be output. Outputting the processing results output by all task execution modules makes the entire task execution process clear and enhances interpretability.

[0060] The electronic device provided in this disclosure receives query requests for the original video through a processor. Before generating query results based on the original video, a decomposition step is added to the query request. That is, the received query request is decomposed into multiple tasks to be executed sequentially, constructing an ordered and executable task sequence. By decomposing the complex query request in advance, the information required for each step of the task execution can be accurately located, reducing the process of indiscriminate global analysis of the original video and reducing the computation on invalid frames from the source, thereby effectively reducing the overall consumption of computing resources. Furthermore, after obtaining the task sequence corresponding to the query request, the task execution modules adapted to each task are called sequentially according to the task sequence to perform unidirectional and multi-dimensional targeted analysis and processing on the original video, obtaining the independent processing results corresponding to each task. It is not necessary to traverse all redundant frames of the video, nor is it necessary to repeatedly calculate the global spatiotemporal features of the video. At the same time, relying on the task decomposition and multi-dimensional targeted video processing mode, the analysis requirements of each decomposed task can be matched, avoiding the defects of coarse sampling, loss of key details caused by global blind analysis, and mismatch of spatiotemporal information in the understanding of long videos in related technologies. Finally, the electronic device determines the query result corresponding to the query request based on the processing results of each task, which not only reduces the consumption of computing resources, but also improves the accuracy of the query result.

[0061] Figure 3 This is a second schematic flowchart illustrating the processor implementing a video-based query method in one embodiment of this disclosure. Figure 3 As shown, in some embodiments, when the processor executes computer program instructions to decompose the query request, it is specifically used to implement S221.

[0062] S221. Input the query request and the constraints of the candidate task execution modules into the large language model. Using the large language model, decompose the query request according to the constraints of the candidate task execution modules to obtain the task sequence.

[0063] In some embodiments, upon receiving a query request, the query request is decomposed into a task sequence in one go using a large language model under the constraints of candidate task execution modules. The task sequence includes multiple tasks executed sequentially, each task corresponding to at least one task executable by a task execution module. For ease of understanding, the task sequence can be viewed as a unidirectional execution path (without loop operations), and each path node on this execution path corresponds to a task execution module.

[0064] In some embodiments, the query request and the constraints of the candidate task execution modules are input together into the large language model. The large language model then decomposes the query request based on the constraints of the candidate task execution modules. Compared to inputting the constraints of the candidate task execution modules into the large language model in advance, this embodiment reduces the training steps of the large language model and the fine-tuning of its parameters.

[0065] In some embodiments, prompt words are input into a large language model, which then decomposes the query request based on the constraints of the candidate task execution module.

[0066] In some embodiments, the large language model is deployed on an electronic device. The processor executes computer program instructions to invoke and run the large language model.

[0067] In some embodiments, the large language model is deployed on a cloud server outside of an electronic device. The processor executes computer program instructions to invoke and run the large language model on the cloud server.

[0068] The electronic device provided in this embodiment uses a large language model combined with the constraints of each candidate task execution module to complete query decomposition. On the one hand, it relies on the deep semantic understanding and logical reasoning capabilities of the large language model to fully parse the multi-layered requirements of complex query requests. On the other hand, it combines the constraints of the candidate task execution modules to autonomously generate the dependencies between tasks, generating a logically coherent and unidirectionally executed ordered task sequence, reducing the risk of task disorder and improving the accuracy of query results.

[0069] Figure 4 This is the third flowchart illustrating the implementation of a video-based query method by a processor in one embodiment of this disclosure. Figure 4 As shown, in some embodiments, the constraints of the candidate task execution modules include the functions and dependencies of each candidate task execution module; when the processor executes computer program instructions to implement the query request decomposition of the large language model, it is specifically used to implement S2211~S2214.

[0070] S2211. Extract the query elements from the query request.

[0071] The query elements are clues provided by the query request, and can include object query elements. Optionally, the query request can also include time query elements, audio query elements, or visual query elements.

[0072] In some embodiments, a large language model is used to extract query elements from a query request based on data in a knowledge base. In this embodiment, the large language model is used to retrieve data from the knowledge base according to the query request to determine the query elements. Specifically, the query request is encoded to obtain query codes; the similarity between the query codes and the data in the knowledge base is calculated; a subset of query codes whose similarity to the data in the knowledge base is greater than a preset similarity threshold is selected; and the content corresponding to the subset of query codes in the query request is determined as the query elements.

[0073] In one example, the query request is "How are the bottle caps placed at the beginning of the video?" Extracting the query elements from the query request yields the following elements: the time element "at the beginning of the video", the object element "bottle caps", and the visual element "How are these bottle caps placed?"

[0074] S2212. Based on the functions of each candidate task execution module, determine multiple target task execution modules corresponding to the query elements.

[0075] In some embodiments, the functionality of the candidate task execution modules specifies the capabilities and intended use of each module. For example, the candidate task execution module ASR_Search searches for dialogue or narrative cues in audio transcription, the candidate task execution module CLIP_Retrieve retrieves key image frames based on object query elements extracted from the query request, and the candidate task execution module VLM_QA answers visual questions based on selected key image frames. Because the functions of each candidate task execution module differ, the task execution module used for different types of query elements in the query request will also differ. Based on the functions of each candidate task execution module, multiple target task execution modules corresponding to the query elements can be determined.

[0076] In one example, the candidate task execution module CLIP_Retrieve is used to retrieve key image frames based on object query features extracted from a query request. These query features include object query elements. CLIP_Retrieve is then identified as the target task execution module corresponding to the object query elements.

[0077] It is understandable that a query request may contain multiple query elements, and therefore the target task execution modules corresponding to the query elements in the query request may contain multiple modules.

[0078] S2213. Determine the dependencies between multiple target task execution modules based on the dependencies between each candidate task execution module.

[0079] In some embodiments, dependency rules between candidate task execution modules are obtained, specifying whether the candidate task execution modules can collaborate. For example, heavy vision computing modules such as VLM_QA and OBJ_Count cannot be root nodes because they require discrete frames as input. They should depend on upstream frame filtering modules, such as CLIP_Retrieve or Uniform_Sample. Similarly, since Temporal_Crop outputs a time window rather than image frames, it should be further processed by Uniform_Sample or CLIP_Retrieve before being used by vision-based task execution modules. The dependency rules between candidate task execution modules characterize the dependencies between them. Furthermore, based on the dependencies between candidate task execution modules, the dependencies between multiple target task execution modules can be determined.

[0080] In one example, the dependencies between candidate task execution modules include candidate task execution module VLM_QA depending on candidate task execution module CLIP_Retrieve.

[0081] S2214. Generate a task sequence based on the dependencies between multiple target task execution modules.

[0082] In some embodiments, a task sequence is generated based on the dependencies between multiple target task execution modules. The task sequence includes multiple tasks executed sequentially, each task corresponding to at least one task execution module. Different task execution modules can process corresponding subquery requests. The subquery request includes the query elements from the received query request.

[0083] It should be noted that when executing a task in the task sequence, the corresponding task execution module needs to be called. The task execution module processes the video to obtain the processing result corresponding to the task.

[0084] In some embodiments, a data format conversion task can also be performed during the intervals between invoking the task execution module corresponding to the task. The input and output data formats of each target task execution module can be preset. The input and output data formats of the target task execution module can specify the data format of the subquery request, the data format of the upstream output data when inputting into the target task execution module, and the data format of the output data of the target task execution module. For example, the subquery request of the target task execution module CLIP_Retrieve should be a purely visual description containing specific object query elements, rather than time information. Furthermore, the output of the target task execution module CLIP_Retrieve is a list of frame indexes.

[0085] In the embodiments of this disclosure, a data format conversion task is performed between the invocation of the task execution modules corresponding to each task, ensuring that multiple target task execution modules corresponding to the task sequence can execute their tasks smoothly and successfully output processing results. For each target task execution module, its input and output data formats can be preset. Standardized input and output data formats ensure that each target task execution module returns concise and structured processing results, rather than lengthy free text. The highly structured output of intermediate nodes guarantees high accuracy in the reasoning process of the large language model.

[0086] The electronic device provided in this disclosure generates multiple tasks to be executed sequentially based on the query elements in the query request and the functions and dependencies of each candidate task execution module. This is a one-time compilation unidirectional path for generating the query request, which eliminates the "illusion" of the large language model and error accumulation caused by the continuous accumulation of context history in multiple rounds of the traditional method, thereby improving the accuracy of the query results.

[0087] Figure 5 This is the fourth flowchart illustrating the implementation of a video-based query method by a processor in one embodiment of this disclosure. Figure 5 As shown, in some embodiments, when the processor executes computer program instructions to obtain the processing result of the task, it is specifically used to implement S231~S232.

[0088] S231. Obtain the audio and video data files corresponding to the original video. The audio and video data files include audio files and video files. The audio file is the file corresponding to the audio data of the original video, and the video file is the file corresponding to the image data of the original video.

[0089] In some embodiments, before receiving a query request for the original video, the original video is received and stored in memory. Furthermore, based on the original video, an audio-video data file corresponding to the original video is generated. The audio-video data file includes an audio file and a video file; the audio file is the file corresponding to the audio data of the original video, and the video file is the file corresponding to the image data of the original video. The audio file is stored in an audio database, and the video file is stored in a visual database.

[0090] In some embodiments, a visual encoder using a Contrastive Language-Image Pre-training (CLIP) model is used to extract feature vectors from image frames in the original video, and a video file is generated based on the feature vectors of the image frames.

[0091] In some embodiments, structured audio records are generated using Voice Activity Detection (VAD), Automatic Speech Recognition (ASR), and a log of the speaker. These audio records may include, but are not limited to, subtitle text and corresponding sound start and end timestamps. An audio file containing the audio records is then generated.

[0092] In one example, the source of the sound can be an object, a person, or an animal.

[0093] In some embodiments, after obtaining the task sequence, the audio file corresponding to the original video is obtained from the audio database, and the video file corresponding to the original video is obtained from the visual database.

[0094] S232. According to the task order of multiple tasks in the task sequence, call the target task execution module corresponding to each task in turn to process the audio and video data files and obtain the processing results of each task.

[0095] In some embodiments, the target task execution tool module corresponding to each task is called sequentially according to the task order of multiple tasks in the task sequence to process the audio file and / or video file and obtain the processing result of each task.

[0096] In one example, the target task execution module ASR_Search is used to search for dialogue or narrative clues in audio transcription. The target task execution module ASR_Search is invoked to process the audio file, which includes subtitle text. Based on a subquery request, ASR_Search searches the subtitle text to obtain either keyword subtitle text or timestamp information.

[0097] In another example, the target task execution module CLIP_Retrieve is used to retrieve video frames. Calling CLIP_Retrieve retrieves the video file and identifies the key image frames corresponding to the time query elements.

[0098] The electronic device provided in this embodiment first processes the original video to prepare the corresponding audio and video data files, namely the files corresponding to the audio data and the image data of the original video. Then, in the process of determining the query results, it is not necessary to process the entire original video repeatedly. The audio and video data files and image data files corresponding to the original video can be processed separately, which improves the execution efficiency of the task and further improves the efficiency of determining the query results.

[0099] In some embodiments, the target task execution module includes a time-determining task execution module, a retrieval task execution module, and an identification task execution module; when the processor executes computer program instructions to obtain the processing result of the task, it is specifically used to implement: The time determination task execution module is invoked to determine the time window corresponding to the time query element based on the time query element corresponding to the query request. The retrieval task execution module is invoked to search the video file and determine the key image frames corresponding to the time window; The recognition task execution module is invoked to perform object recognition processing on key image frames based on the object query elements in the query request, and the recognition results are obtained.

[0100] In some embodiments, when the processor executes computer program instructions to determine a time window, it specifically implements: calling the time determination task execution module to determine the time window corresponding to the time query element based on the time query element in the query request.

[0101] In some embodiments, when the processor executes computer program instructions to determine the time window, it specifically implements: when no time query element is extracted from the query request, determining the time query element based on the object query element in the query request.

[0102] In this embodiment, the audio file includes audio records corresponding to the audio segment information of the original video. The audio records include subtitle text, the speaker identifier corresponding to the subtitle text, and the start and end timestamp information of the sound corresponding to the subtitle text. When the processor executes computer program instructions to obtain the processing results of the task, it is specifically used to implement: The time determination task execution module is invoked. Based on the object query elements in the query request, the audio records are retrieved to determine the start and end timestamp information of the target sound corresponding to the object query elements. Based on the start and end timestamp information of the target sound, the time query elements are determined.

[0103] Specifically, the object query elements may include sound object query elements and / or subtitle text query elements. Based on the sound object identifier query elements and / or subtitle text query elements, the audio records are retrieved to determine the start and end timestamp information of the target sound corresponding to the object query elements.

[0104] In some embodiments, the time determination task execution module includes a first sub-time determination task execution module ASR_Search and a second sub-time determination task execution module Temporal_Crop. When the query elements in the query request include at least one of subtitle text query elements and sound object query elements, the first sub-time determination task execution module ASR_Search is invoked to retrieve audio records based on the query elements in the query request, determining the start and end timestamp information of the target sound corresponding to the query elements; the second sub-time determination task execution module Temporal_Crop is invoked to determine the time window corresponding to the start and end timestamp information of the target sound.

[0105] In this embodiment, the first sub-time determination task execution module ASR_Search, upstream of the second sub-time determination task execution module Temporal_Crop, provides the start and end timestamp information of the target sound, i.e., the time interval. It then calls the second sub-time determination task execution module Temporal_Crop to convert the time interval into a standardized time list. For example, the second sub-time determination task execution module Temporal_Crop ultimately outputs a normalized time window [start, end] in seconds.

[0106] The electronic device provided in this disclosure can determine the time query element based on the query elements other than the time query element in the query request. In this way, even if no time clue is provided in the query request, the corresponding time interval can be determined based on other clues. The time interval can be obtained first, and the query result can be determined after cropping key video frames based on the time interval. This mechanism enables compatibility with open natural language queries without time constraints and can broaden the applicable scope of long video retrieval scenarios.

[0107] In some embodiments, the CLIP_Retrieve module is invoked to retrieve video files and determine key image frames corresponding to time windows. In this embodiment, the video file includes feature vectors of each frame in the original video. When the processor executes computer program instructions to determine key image frames, it specifically performs the following: encoding the object query elements in the query request using the CLIP text encoder to obtain query element vectors; calculating the similarity between the feature vectors of each frame corresponding to the time window and the query element vectors; and selecting the image frames corresponding to the feature vectors of frames with similarity greater than a preset similarity as key image frames.

[0108] In some embodiments, the video file includes feature vectors of each frame in the original video; when the processor executes computer program instructions to determine key image frames, it specifically implements: Call the retrieval task execution module to calculate the similarity between the feature vector of each frame within the time window and the object query elements in the query request; Based on similarity, the image frames corresponding to the feature vectors of each frame within the time window are sorted in descending order to obtain the frame sequence. The first K target image frames in the frame sequence are identified as key image frames, where K is a positive integer.

[0109] The feature vectors of each frame in the video file are obtained by sampling and encoding the original video according to a preset sampling rate.

[0110] The electronic device provided in this embodiment not only selects multiple key image frames that are highly similar to the object query elements in the query request, but also limits the number of key image frames. This reduces the amount of computation when the recognition task execution module is called to perform object recognition, thereby reducing the consumption of computing resources while ensuring high accuracy of the query results.

[0111] In some embodiments, the recognition task execution module includes a visual question answering task execution module (VLM_QA), which integrates a pre-trained visual language model. When the processor executes computer program instructions to implement object recognition, it specifically performs the following: calls the visual question answering task execution module (VLM_QA), utilizes the pre-trained visual language model integrated in the VLM_QA, performs object recognition processing on key image frames, and obtains the recognition result.

[0112] In some embodiments, the visual language model is deployed on an electronic device. The processor executes computer program instructions to invoke and run the visual language model, perform object recognition processing on key image frames, and obtain recognition results.

[0113] In some embodiments, the visual language model is deployed on a cloud server outside the electronic device. The processor executes computer program instructions to invoke the visual language model on the cloud server, run the visual language model, perform object recognition processing on key image frames, and obtain recognition results.

[0114] Among them, the Vision-Language Model (VLM) can be a multimodal neural network model that integrates computer vision and natural language processing.

[0115] It should be noted that the visual question answering task execution module VLM_QA can process the corresponding subquery requests and obtain the processing results of the subquery requests. The subquery requests include the object query elements from the received query requests.

[0116] In one example, if the query request is "how are the bottle caps placed at the beginning of the video", then the subquery request corresponding to the visual question answering task execution module VLM_QA is "how are these bottle caps arranged".

[0117] In some embodiments, on edge devices with limited computing power, the Visual Question Answering Task Execution Module (VLM_QA) can replace the Visual Language Model (VLM) with a lightweight large visual model with fewer parameters or a dedicated small object detection model.

[0118] In some embodiments, when the query request includes a counting query element, the processor executes computer program instructions to implement object recognition, and is further configured to: call the recognition task execution module to identify the number of target objects in the key image frame based on the counting query element in the query request, and obtain the recognition result.

[0119] In this embodiment, the identification task execution module includes a quantity identification task execution module OBJ_Count. The processor executes computer program instructions specifically to: call the quantity identification task execution module OBJ_Count, identify the quantity of target objects in the key image frame based on the counting query elements in the query request, and obtain the identification result.

[0120] It should be noted that the quantity recognition task execution module OBJ_Count can process the corresponding subquery requests and obtain the processing results. The subquery request is the counting query element in the received query request. The counting query element can be the counting query element of the target object.

[0121] In one example, the query request is "How many bottle caps are there at the beginning of the video?". In this query request, the target object is bottle caps, and the count query element is "How many are there?".

[0122] The electronic device provided in this disclosure can meet the user's counting needs and improve the user experience by using the counting query elements in the query request.

[0123] The electronic device provided in this disclosure has a clear time clipping and retrieval path planned before execution, so it can accurately locate the key segments of the video and perform a secondary retrieval of key image frames within the specified key segments. Through this serial execution design, not only can the retrieval of invalid image frames be reduced and the computational overhead be reduced, but the key image frames can also be determined step by step and object recognition can be performed through this design. The accuracy of the query results is improved through spatiotemporal linkage.

[0124] Figure 6 This is the fifth flowchart illustrating the implementation of a video-based query method by a processor in one embodiment of this disclosure. Figure 6As shown, in some embodiments, the processor executes computer program instructions to generate a video file before processing the audio and video data file, and also implements S310 to S330.

[0125] S310. Sample the original video to obtain multiple first image frames.

[0126] In some embodiments, the original video is uniformly sampled according to a preset sampling rate to obtain multiple first image frames. For example, the preset sampling rate is 1 (FPS), that is, one frame is captured every second.

[0127] S320. Encode multiple first image frames to obtain the feature vector of each frame.

[0128] In some embodiments, the CLIP visual encoder is used to encode each of the multiple first image frames to obtain the feature vector of each frame.

[0129] In one example, the CLIP visual encoder is used to encode the first image frame, generating a 512-dimensional embedding vector for the first image frame. This 512-dimensional embedding vector is then determined as the feature vector of the first image frame.

[0130] In some embodiments, a multimodal feature extraction model (e.g., ImageBind, VideoMAE) is used to encode multiple first image frames to obtain feature vectors for each frame.

[0131] S330: Generate video files based on the feature vectors of each frame.

[0132] In some embodiments, a video file including feature vectors of each frame is generated according to predefined file generation rules.

[0133] The electronic device provided in this disclosure generates a video file in advance to retrieve key image frames. During task execution, the video file can be used directly to retrieve key image frames without having to process the entire original video repeatedly, thus improving the retrieval efficiency of key image frames.

[0134] Figure 7 This is a sixth flowchart illustrating the implementation of a video-based query method by a processor in one embodiment of this disclosure. Figure 7 As shown, in some embodiments, the processor executes computer program instructions to generate an audio file before processing the audio and video data file, and also implements S410 to S430.

[0135] S410. Sample the audio stream in the original video to obtain audio information.

[0136] In some embodiments, the presence or absence of an audio track in the original video is detected to obtain a detection result. If the detection result indicates that the original video includes an audio track, the audio stream in the original video is sampled to obtain audio information. That is, the existence of an audio track is verified first, and then a decision is made on whether to perform sampling processing to avoid invalid sampling.

[0137] In one example, the audio / video metadata detection tool ffprobe is used to verify the existence of an audio track.

[0138] In another example, for the original video, the audio stream is extracted and downsampled to a preset audio sampling rate to obtain the audio information. For example, the preset audio sampling rate is 16kHz.

[0139] S420. Extract at least one speech segment from the audio information.

[0140] In some embodiments, the speech information of a speaking object is separated from the audio information to obtain at least one speech segment. There can be one or more speaking objects. If there are multiple speaking objects, corresponding speech segment information can be obtained for each speaking object.

[0141] In some embodiments, speech activity detection (VAD) is used to detect speech activity from audio information and provide signal-level hard boundaries for speech, making temporal localization less susceptible to semantic drift. Based on the signal-level hard boundaries, at least one speech segment is extracted from the audio information.

[0142] S430. Generate an audio file based on at least one speech segment.

[0143] In some embodiments, at least one speech segment is denoised to obtain speech segment information corresponding to the speaker. An audio file is then generated based on the speech segment information corresponding to the speaker.

[0144] In some embodiments, if the audio information includes speech information, then at least one speech segment is transcribed using automatic speech recognition (ASR) to generate an audio file.

[0145] The electronic device provided in this disclosure can pre-generate an audio file corresponding to the original video before receiving a query request, and use it during task execution. As an example, the audio file can be used to determine the time window. As another example, the audio file can also be used to retrieve the corresponding video segment, thereby improving query efficiency.

[0146] Figure 8 This is the seventh flowchart illustrating the implementation of a video-based query method by a processor in one embodiment of this disclosure. Figure 8As shown, in some embodiments, when the processor executes computer program instructions to generate an audio file, it specifically implements S431 to S433.

[0147] S431. For each speech segment in at least one speech segment information, transcribe the speech segment information to obtain the subtitle text and the corresponding audio start and end timestamp information.

[0148] In some embodiments, if the audio information includes speech information, automatic speech recognition (ASR) is used to transcribe each speech segment to obtain subtitle text for each speech segment. ASR is then used to map the speech segment information corresponding to the subtitle text back to the timeline of the original video to obtain the start and end timestamp information of the audio corresponding to the subtitle text.

[0149] For example, in a driving scenario, if a user's query request is "find video clips where the conversation mentions accident compensation", the content of the conversation cannot be known solely from the video footage. Therefore, it is necessary to transcribe each audio clip to obtain subtitle text for each audio clip and bind the corresponding start and end timestamps of the audio to locate the video clip based on the subtitle text and perform video retrieval.

[0150] It should be noted that, since each speech segment is short and time-limited, the relative word-level timestamps generated by automatic speech recognition (ASR) can be accurately mapped back to the video timeline, thereby reducing timestamp drift.

[0151] S432. Based on the subtitle text and the corresponding audio start and end timestamp information, generate audio records, with each audio record corresponding to one speech segment.

[0152] In some embodiments, the audio recording is a structured recording. For each audio segment, there is a mapping relationship between its subtitle text and the corresponding audio start and end timestamp information. Based on the mapping relationship between the subtitle text and the corresponding audio start and end timestamp information, an audio recording is generated, with one audio recording corresponding to one audio segment.

[0153] In one example, the audio records are presented as a table, with each row containing one audio record and a subtitle text for each audio segment, along with the start and end timestamps of the corresponding audio.

[0154] S433. Generate an audio file based on the audio record corresponding to at least one speech segment information.

[0155] In some embodiments, an audio file is generated, which includes an audio record corresponding to at least one speech segment, according to predefined file generation rules.

[0156] The electronic device provided in this embodiment first obtains the subtitle text corresponding to at least one audio segment and the start and end timestamp information of the audio corresponding to the subtitle text, then generates the corresponding audio record, and generates an audio file based on the audio record. This achieves accurate binding between subtitle semantics and audio timing, which can reduce the probability of mismatch across time periods when performing tasks in the future, thereby improving the accuracy of query results.

[0157] In some embodiments, the audio recording also includes an identifier for the speaker corresponding to the subtitle text. When the processor executes computer program instructions to generate an audio file, it is also used to: For each speech segment in at least one speech segment information, extract the voiceprint features of the speaker in the speech segment information; Cluster the voiceprint features of the speaker from at least one speech segment to obtain at least one voiceprint category; Assign a voice-generating object identifier to each voiceprint category; Based on the voiceprint identifier of the voiceprint object in at least one voiceprint category and the voiceprint features of the voiceprint object in the speech segment information, determine the voiceprint identifier in the audio record corresponding to the speech segment information.

[0158] In this embodiment, a voice object verification encoder is used to extract the voiceprint features of the voice object from the speech segment information. For at least one voiceprint feature of the voice object, cosine distance is used to perform agglomerative hierarchical clustering (AHC) to obtain at least one voiceprint category. A voice object identifier is assigned to each voiceprint category; based on the voice object identifier corresponding to each voiceprint category, the voice object identifier of the speech segment information corresponding to the voiceprint features of the voice object within each voiceprint category can be determined. The voice object identifier is recorded in the corresponding position in the audio record corresponding to the speech segment information, so that there is a mapping relationship between the subtitle text and the voice object identifier.

[0159] In one example, the voice source is identified as the voice source identifier (ID).

[0160] The electronic device provided in this disclosure matches a corresponding speaker identifier to each voice segment and records it in an audio recorder. When a user's query request involves a speaker, the speaker identifier can be used to quickly pinpoint the user's query needs. Furthermore, a time window can be determined based on the speaker identifier, which can improve the retrieval efficiency of key image frames.

[0161] Figure 9 This is the eighth flowchart illustrating the implementation of a video-based query method by a processor in one embodiment of this disclosure. Figure 9 As shown, in some embodiments, when the processor executes computer program instructions to determine the query result, it is specifically used to implement S241~S242.

[0162] S241. According to the calling order of the execution modules of each target task, the processing results of each task are spliced ​​together to obtain a text evidence chain.

[0163] The text evidence chain is a collection of texts obtained by concatenating the processing results of each task according to the calling order of the execution modules of each target task. This text collection completely records the output results of each execution module of the target task.

[0164] It should be noted that due to limitations in the input and output data formats of the target task execution modules, the processing results of these modules are highly structured and concise information (redundant and long texts have been removed). By concatenating the processing results of each task according to the order in which they are called, a short textual evidence chain can be obtained.

[0165] In some embodiments, the processing results of each task include tags of the task execution modules. Based on the tags in the processing results of each task, and in accordance with the calling order of the execution modules of each target task, the processing results of each task are concatenated to obtain a text evidence chain.

[0166] S242. Based on the textual evidence chain, determine the first query result corresponding to the query request.

[0167] In some embodiments, the processing result output by the last target task execution module in the text evidence chain is determined as the first query result corresponding to the query request.

[0168] In some embodiments, the textual evidence chain is determined as the first query result corresponding to the query request.

[0169] The electronic device provided in this disclosure can clearly present the complex reasoning process of the entire video by splicing the processing results of the multimodal task execution module according to the calling order of each task execution module. For example, it is clear which step cut the time and which step retrieved the image, and it has high interpretability.

[0170] In some embodiments, when the processor executes computer program instructions to determine the query results, it specifically implements the following: The query request and the text evidence chain are input into the large language model, and the large language model is used to find the first query result corresponding to the query request from the text evidence chain.

[0171] In this embodiment, the query request and text evidence chain are input into the aggregated large language model (LLM_AGGREGATE), and the aggregated large language model is used to perform the final logical induction and output the first query result corresponding to the query request.

[0172] The electronic device provided in this disclosure reduces the risk of illusion from long texts and improves the accuracy of query results by inputting a short text evidence chain into a large language model and using the large language model to find the first query result corresponding to the query request from the text evidence chain.

[0173] In some embodiments, when the processor executes computer program instructions to determine the query result, it is also used to: In response to the first query result not meeting the preset conditions, the original video is subjected to frame extraction processing to obtain multiple second image frames; The query request and multiple second image frames are input into a large language model. The large language model is then used to recognize and process the multiple second image frames according to the query request to obtain the second query result.

[0174] In some embodiments, preset conditions can be pre-set according to actual needs. For example, preset conditions include not obtaining valid results or receiving the user's target input information. The target input information indicates that the user is dissatisfied with the query results.

[0175] In this embodiment, the similarity between the first query result and the preset query result is calculated. If the similarity between the first query result and the preset query result is greater than a preset threshold, the original video is processed by frame extraction to obtain multiple second image frames. The query request and the multiple second image frames are input into a large language model. Using the large language model, the multiple second image frames are processed for recognition according to the query request to obtain the second query result.

[0176] The default query result is invalid.

[0177] Alternatively, the similarity between the user input information received after outputting the first query result and the target input information can be calculated. If the similarity between the user input information and the target input information is greater than a preset threshold, the original video is processed by frame extraction to obtain multiple second image frames. The query request and the multiple second image frames are then input into a large language model. Using the large language model, the multiple second image frames are processed for recognition according to the query request to obtain the second query result.

[0178] In one example, the default query result is predefined. For example, the default query result is "None of the above".

[0179] In another example, the target input is predefined. For instance, the target input could be "The result is not what I wanted".

[0180] It should be noted that both the preset query results and the target input information can include multiple results.

[0181] In some embodiments, in response to the first query result not meeting the preset conditions, a preset number of uniform frame extraction processes are performed on the original video to obtain a preset number of second image frames; the query request and the preset number of second image frames are input into a large language model, and the large language model is used to perform recognition processing on the preset number of second image frames according to the query request to obtain a second query result.

[0182] In some embodiments, the large language model is deployed on an electronic device. The processor executes computer program instructions to invoke and run the large language model.

[0183] In some embodiments, the large language model is deployed on a cloud server outside of an electronic device. The processor executes computer program instructions to invoke and run the large language model on the cloud server.

[0184] The electronic device provided in this embodiment triggers a fallback mechanism when it cannot obtain the query results required by the user. It directly performs frame extraction on the original video and directly prompts the Visual Large Model (VLM) to answer the query request, so as to ensure the minimum performance limit of the electronic device. The robustness of the electronic device is improved by the dual-standard global fallback method.

[0185] In the above embodiments, the electronic device can be implemented using a loop-free, unidirectional execution agent architecture, or it can rely on a deep learning framework (PyTorch).

[0186] Exemplary methods This disclosure also provides a video-based query method, which can be executed by an electronic device or a processor, and includes: Receive query requests for the original video; The query request is decomposed to obtain a task sequence, which includes multiple tasks to be executed sequentially. According to the task order of multiple tasks in the task sequence, the target task execution module corresponding to each task is called in turn to process the original video and obtain the processing results of each task. Based on the processing results of each task, determine the first query result corresponding to the query request.

[0187] In some embodiments, decomposing a query request into a task sequence may include: The query request and the constraints of the candidate task execution modules are input into the large language model. The large language model is then used to decompose the query request according to the constraints of the candidate task execution modules to obtain the task sequence.

[0188] In some embodiments, the constraints of the candidate task execution modules include the functionality and dependencies of each candidate task execution module; based on the constraints of the candidate task execution modules, the query request is decomposed to obtain a task sequence, which may include: Extract the query elements from the query request; Based on the functions of each candidate task execution module, determine multiple target task execution modules corresponding to the query elements; Based on the dependencies between the execution modules of each candidate task, the dependencies between the execution modules of multiple target tasks are determined. A task sequence is generated based on the dependencies between multiple target task execution modules.

[0189] In some embodiments, the target task execution module corresponding to each task is called sequentially according to the task order of multiple tasks in the task sequence to process the original video and obtain the processing results of each task, which may include: Obtain the audio and video data files corresponding to the original video. The audio and video data files include audio files and video files. The audio file is the file corresponding to the audio data of the original video, and the video file is the file corresponding to the image data of the original video. Following the task order of multiple tasks in the task sequence, the target task execution tool module corresponding to each task is called sequentially to process the audio and video data files and obtain the processing results of each task.

[0190] In some embodiments, the target task execution module includes a time determination task execution module, a retrieval task execution module, and an identification task execution module; it calls the target task execution tool module corresponding to each task to process the audio and video data files and obtain the processing results of each task, which may include: The time determination task execution module is invoked to determine the time window corresponding to the time query element based on the time query element corresponding to the query request. The retrieval task execution module is invoked to search the video file and determine the key image frames corresponding to the time window; The recognition task execution module is invoked to perform object recognition processing on key image frames based on the object query elements in the query request, and the recognition results are obtained.

[0191] In some embodiments, the audio file includes audio records corresponding to speech segments of the original video, and the audio records include subtitle text, a speaker identifier corresponding to the subtitle text, and sound start and end timestamp information corresponding to the subtitle text; calling the time determination task execution module to determine the time window corresponding to the time query element based on the time query element corresponding to the query request may include: The time determination task execution module is invoked. Based on the object query elements in the query request, the audio records are retrieved to determine the start and end timestamp information of the target sound corresponding to the object query elements. Based on the start and end timestamp information of the target sound, the time query elements are determined.

[0192] In some embodiments, the video file includes feature vectors of each frame in the original video; invoking the retrieval task execution module to retrieve the video file and determine the key image frames corresponding to the time window may include: Call the retrieval task execution module to calculate the similarity between the feature vector of each frame within the time window and the object query elements in the query request; Based on similarity, the image frames corresponding to the feature vectors of each frame within the time window are sorted in descending order to obtain the frame sequence. The first K target image frames in the frame sequence are identified as key image frames, where K is a positive integer.

[0193] In some embodiments, the recognition task execution module is invoked to perform object recognition processing on key image frames based on object query elements in the query request, and the recognition result may include: The recognition task execution module is invoked to identify the number of target objects in the key image frame based on the counting query elements in the query request, and the recognition result is obtained.

[0194] In some embodiments, the method may further include: The original video is sampled to obtain multiple first image frames; Encode multiple first image frames to obtain the feature vectors of each frame; Video files are generated based on the feature vectors of each frame.

[0195] In some embodiments, the method may further include: The audio stream in the original video is sampled and processed to obtain audio information; Extract at least one speech segment from the audio information; Generate an audio file based on at least one speech segment.

[0196] In some embodiments, generating an audio file based on at least one speech segment may include: For each audio segment in at least one audio segment information, the audio segment information is transcribed to obtain the subtitle text and the corresponding audio start and end timestamp information; Based on the subtitle text and the corresponding audio start and end timestamp information, an audio record is generated, with one audio record corresponding to one speech segment. An audio file is generated based on the audio record corresponding to at least one speech segment.

[0197] In some embodiments, the method may further include: For each speech segment in at least one speech segment information, extract the voiceprint features of the speaker in the speech segment information; Cluster the voiceprint features of the speaker from at least one speech segment to obtain at least one voiceprint category; Assign a voice-generating object identifier to each voiceprint category; Based on the voiceprint identifier of the voiceprint object in at least one voiceprint category and the voiceprint features of the voiceprint object in the speech segment information, determine the voiceprint identifier in the audio record corresponding to the speech segment information.

[0198] In some embodiments, determining the first query result corresponding to the query request based on the processing results of each task may include: According to the calling order of the execution modules of each target task, the processing results of each task are spliced ​​together to obtain the text evidence chain; Based on the textual evidence chain, determine the first query result corresponding to the query request.

[0199] In some embodiments, determining the first query result corresponding to the query request based on the textual evidence chain may include: The query request and the text evidence chain are input into the large language model, and the large language model is used to find the first query result corresponding to the query request from the text evidence chain.

[0200] In some embodiments, the method may further include: In response to the first query result not meeting the preset conditions, the original video is subjected to frame extraction processing to obtain multiple second image frames; The query request and multiple second image frames are input into a large language model. The large language model is then used to recognize and process the multiple second image frames according to the query request to obtain the second query result.

[0201] The query request determination method provided in the above embodiments can be applied to various scenarios, including but not limited to real-time understanding of streaming long videos and offline long video understanding scenarios.

[0202] The specific principles, implementation methods, and corresponding technical effects of each step in the above-mentioned query request determination method can be found in the description of the electronic device embodiment. For the sake of brevity, they will not be elaborated here.

[0203] Exemplary computer program products and computer-readable storage media In addition to the methods and apparatus described above, embodiments of this disclosure may also provide a computer program product, including computer program instructions that, when executed by a processor, cause the processor to perform the steps of the query request determination method described in the various embodiments of this disclosure in the "Exemplary Methods" section above.

[0204] Computer program products can be written in any combination of one or more programming languages ​​to perform the operations of embodiments of this disclosure. These programming languages ​​include object-oriented programming languages ​​such as Java and C++, as well as conventional procedural programming languages ​​such as C or similar languages. The program code can be executed entirely on a user's computing device, partially on a user's computing device, as a standalone software package, partially on a user's computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0205] Furthermore, embodiments of this disclosure may also be computer-readable storage media storing computer program instructions thereon, which, when executed by a processor, cause the processor to perform the steps of the query request determination method of the various embodiments of this disclosure described in the "Exemplary Methods" section above.

[0206] Computer-readable storage media may take the form of any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may include, but is not limited to, systems, apparatuses, or devices that are electrical, magnetic, optical, electromagnetic, infrared, or semiconductor, or any combination thereof. More specific examples of readable storage media (a non-exhaustive list) include: electrical connections having one or more wires; portable disks; hard disks; random access memory (RAM); read-only memory (ROM); erasable programmable read-only memory (EPROM); or flash memory; optical fiber; compact disc read-only memory (CD-ROM); optical storage devices; magnetic storage devices; or any suitable combination thereof.

[0207] The basic principles of this disclosure have been described above with reference to specific embodiments. However, the advantages, benefits, and effects mentioned in this disclosure are merely examples and not limitations, and should not be considered as essential features of each embodiment of this disclosure. Furthermore, the specific details disclosed above are for illustrative and facilitative purposes only, and are not limitations. These details do not limit the scope of this disclosure to the necessity of employing the aforementioned specific details for implementation.

[0208] Various modifications and variations can be made to this disclosure without departing from its spirit and scope. Therefore, this disclosure is also intended to include such modifications and variations if they fall within the scope of the claims of this disclosure and their equivalents.

Claims

1. An electronic device, characterized in that, include: Processor and memory storing computer program instructions; The processor executes the computer program instructions to achieve: Receive query requests for the original video; The query request is decomposed to obtain a task sequence, which includes multiple tasks to be executed sequentially; According to the task order of multiple tasks in the task sequence, the target task execution module corresponding to each task is called in turn to process the original video and obtain the processing result of each task. Based on the processing results of each of the tasks, a first query result corresponding to the query request is determined.

2. The electronic device according to claim 1, characterized in that, The processor is specifically used to implement: The query request and the constraints of the candidate task execution modules are input into the large language model. Using the large language model, the query request is decomposed according to the constraints of the candidate task execution modules to obtain the task sequence.

3. The electronic device according to claim 2, characterized in that, The constraints of the candidate task execution modules include the functions and dependencies of each candidate task execution module; The processor is specifically used to implement: Extract the query elements from the query request; Based on the functions of each candidate task execution module, multiple target task execution modules corresponding to the query elements are determined; The dependencies between the multiple target task execution modules are determined based on the dependencies between the candidate task execution modules. The task sequence is generated based on the dependencies between the multiple target task execution modules.

4. The electronic device according to claim 1, characterized in that, The processor is specifically used to implement: Obtain the audio and video data file corresponding to the original video. The audio and video data file includes an audio file and a video file. The audio file is the file corresponding to the audio data of the original video, and the video file is the file corresponding to the image data of the original video. According to the task order of multiple tasks in the task sequence, the target task execution tool module corresponding to each task is called in turn to process the audio and video data file and obtain the processing result of each task.

5. The electronic device according to claim 4, characterized in that, The target task execution module includes a time-determining task execution module, a retrieval task execution module, and an identification task execution module; The processor is specifically used to implement: The time determination task execution module is invoked to determine the time window corresponding to the time query element based on the time query element corresponding to the query request. The retrieval task execution module is invoked to retrieve the video file and determine the key image frames corresponding to the time window; The recognition task execution module is invoked to perform object recognition processing on the key image frame based on the object query elements in the query request, and the recognition result is obtained.

6. The electronic device according to claim 5, characterized in that, The audio file includes audio records corresponding to the audio segment information of the original video. The audio records include subtitle text, the speaker identifier corresponding to the subtitle text, and the sound start and end timestamp information corresponding to the subtitle text. The processor is specifically used to implement: The time determination task execution module is invoked to retrieve the audio record based on the object query elements in the query request, and to determine the target sound start and end timestamp information corresponding to the object query elements; Based on the start and end timestamp information of the target sound, the time query element is determined.

7. The electronic device according to claim 5, characterized in that, The video file includes feature vectors of each frame in the original video; The processor is specifically used to implement: The retrieval task execution module is invoked to calculate the similarity between the feature vector of each frame within the time window and the object query elements in the query request. Based on the similarity, the image frames corresponding to the feature vectors of each frame within the time window are sorted in descending order to obtain a frame sequence. The first K target image frames in the frame sequence are identified as key image frames, where K is a positive integer.

8. The electronic device according to claim 5, characterized in that, The processor is also used to implement: The recognition task execution module is invoked to identify the number of target objects in the key image frame based on the counting query elements in the query request, and the recognition result is obtained.

9. The electronic device according to claim 4, characterized in that, The processor is also used to implement: The original video is sampled to obtain multiple first image frames; The plurality of first image frames are encoded to obtain the feature vector of each frame; The video file is generated based on the feature vectors of each frame.

10. The electronic device according to claim 4, characterized in that, The processor is also used to implement: The audio stream in the original video is sampled and processed to obtain audio information; Extract at least one speech segment from the audio information; An audio file is generated based on the at least one speech segment information.

11. The electronic device according to claim 10, characterized in that, The processor is specifically used to implement: For each of the at least one audio segment information, the audio segment information is transcribed to obtain subtitle text and the corresponding audio start and end timestamp information; Based on the subtitle text and the corresponding sound start and end timestamp information, an audio record is generated, and one audio record corresponds to one speech segment information; An audio file is generated based on the audio record corresponding to the at least one speech segment information.

12. The electronic device according to claim 11, characterized in that, The processor is also used to implement: For each of the at least one speech segment information, extract the voiceprint features of the speaker of the speech segment information; Cluster the voiceprint features of the speaker of the at least one speech segment information to obtain at least one voiceprint category; Assign a voice-generating object identifier to each of the aforementioned voiceprint categories; Based on the voiceprint identifier of the at least one voiceprint category and the voiceprint features of the voiceprint of the voiceprint information, the voiceprint identifier in the audio record corresponding to the voiceprint information is determined.

13. The electronic device according to claim 1, characterized in that, The processor is specifically used to implement: According to the calling order of the execution modules of each target task, the processing results of each task are concatenated to obtain a text evidence chain; Based on the textual evidence chain, a first query result corresponding to the query request is determined.

14. The electronic device according to claim 13, characterized in that, The processor is specifically used to implement: The query request and the text evidence chain are input into a large language model, and the large language model is used to find the first query result corresponding to the query request from the text evidence chain.

15. The electronic device according to claim 1, characterized in that, The processor is also used to implement: In response to the first query result not meeting the preset conditions, the original video is subjected to frame extraction processing to obtain multiple second image frames; The query request and the plurality of second image frames are input into a large language model. The large language model is then used to perform recognition processing on the plurality of second image frames according to the query request to obtain a second query result.

16. A video-based query method, characterized in that, include: Receive query requests for the original video; The query request is decomposed to obtain a task sequence, which includes multiple tasks to be executed sequentially; According to the task order of multiple tasks in the task sequence, the target task execution module corresponding to each task is called in turn to process the original video and obtain the processing result of each task. Based on the processing results of each of the tasks, a first query result corresponding to the query request is determined.

17. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer program instructions that, when executed by a processor, implement the video-based query method as described in claim 16.

18. A computer program product, characterized in that, It includes a computer program that, when executed by a processor, implements the video-based query method as described in claim 16.

Images