Video processing method, device and equipment

Abstract

The invention discloses a video processing method, device and equipment, and relates to the technical field of cloud. According to the invention, the video cloud platform is in butt joint with a plurality of video acquisition devices. And the video cloud platform acquires videos acquired by the plurality of video acquisition devices and stores the videos. And the video cloud platform provides an interactive interface for the user. And the video cloud platform receives task description information input by a user through the interactive interface, wherein the task description information comprises part or all of the following modes: texts, voices, images and actions. The video cloud platform analyzes the intention of the task description information and generates a task; and calling the multi-modal large model to execute the task based on the video, and feeding back an execution result of the task to the user. When the number of the video acquisition devices is large, the video cloud platform obtains a large number of videos from the video acquisition devices, and the video cloud platform has the capacity of processing mass videos. The video cloud platform allows a user to input task description information of different modes, and generates and completes a task. And various different requirements of the user are met, and the system is more intelligent and has better user experience.

Description

Technical Field

[0001] The present application relates to the field of cloud technology, and in particular to a video processing method, apparatus and device. Background Art

[0002] With the increase in the scale of camera deployment and its popularization, automatic video recognition has become a mainstream demand. Automatic video recognition relies on manual labor and no longer relies on manual labor, and can identify targets in the video relatively quickly.

[0003] However, automatic video recognition relies on a specific algorithm, which is usually only applied to a small number of recognition instructions. That is, only this recognition instruction can trigger the operation of the algorithm and recognize the video. The expression of this recognition instruction is fixed, which also makes the algorithm only able to recognize specific categories of targets from the video. This type of algorithm is designed according to certain user needs and can only recognize specific categories of targets. It is targeted. This type of algorithm involved in automatic video recognition cannot meet the different needs of users and is relatively mechanical. Summary of the invention

[0004] The embodiments of the present application provide a video processing method, apparatus, and device for meeting different needs of users for video recognition.

[0005] In the first aspect, the embodiment of the present application provides a video processing method, which can be executed by a video cloud platform, which is directly connected to multiple video acquisition devices and can directly interact with the multiple video acquisition devices, such as transmitting videos. The video cloud platform has a built-in multimodal large model, which is a machine learning model that supports information input in multiple modes. In this method, the video cloud platform can obtain videos collected by multiple video acquisition devices and save the videos.

[0006] The video cloud platform provides an interactive interface for users. The interactive interface allows users to interact with the video cloud platform. The interactive interface allows users to input task description information according to their own needs and their own expression habits. The video cloud platform receives the task description information input by the user through the interactive interface. The task description information includes some or all of the following modes: text, voice, image, and action.

[0007] After receiving the task description information, the video cloud platform can analyze the intent of the task description information and generate a task corresponding to the task description information; thereafter, it can call a multimodal large model based on the video to execute the task and provide feedback on the execution results of the task to the user.

[0008] Through the above method, the video cloud platform can directly connect with multiple video acquisition devices, which is convenient for obtaining videos from the video acquisition devices. When the number of video acquisition devices is large, the video cloud platform can also obtain a large number of videos from the video acquisition devices. The video cloud platform has the ability to process massive videos. The video cloud platform is user-oriented, allowing users to input task description information with different modalities, understand the task description information, generate tasks corresponding to the task description information, and complete the task. The types of user needs that the video cloud platform can handle have been expanded to meet the various needs of users. The video cloud platform is more intelligent and has a better user experience.

[0009] In a possible implementation, when the video cloud platform calls a multimodal large model based on a video to perform a task, the target video required to perform the task can be queried from the saved videos; and the task can be performed based on the target video.

[0010] Through the above method, the video cloud platform has visual understanding capabilities, can understand the content displayed in the video and find the target video required to perform the task, effectively ensuring that the task can be completed.

[0011] In a possible implementation, when saving the video, the video cloud platform may directly save the acquired video, or may first process the video and then save the processed video.

[0012] The processing of the video includes some or all of the following: adding content description information to the video, setting scene tags, setting location information and time information for the video, video frame extraction, and short video extraction. The location information is used to describe the location where the video was shot, and the time information is used to describe the shooting time of the video.

[0013] Through the above method, by processing the video, such as adding some information such as content description information, location information and time information, the video can be classified to facilitate the subsequent search for the target video. The method of extracting frames and short videos can effectively reduce the amount of data required to be saved, ensuring that the video cloud platform can reduce data redundancy and save more valid videos.

[0014] In a possible implementation, when the video cloud platform searches for a target video required to perform a task from a saved video, it can search for the target video from the video based on part or all of the video's content description information, scene tags, location information, and time information.

[0015] Through the above method, the video cloud platform can quickly find the target video based on some information of the video, further reducing the time consumption of task execution and improving the efficiency of task execution.

[0016] In a possible implementation, when the video cloud platform calls a multimodal large model to perform a task based on a video, the task can be decomposed into multiple subtasks; based on the video, the multimodal large model is called to perform multiple subtasks.

[0017] Through the above method, task decomposition can decompose some complex tasks into multiple simple subtasks, and the execution of subtasks is faster, further ensuring the execution efficiency of the task.

[0018] In one possible implementation, the multimodal big model is connected to at least one industry model. When the video cloud platform calls the multimodal big model to perform a task based on a video, it can call the multimodal big model and at least one industry model to perform tasks or subtasks (corresponding to the scenario of decomposing the task).

[0019] Through the above method, the industry model is a more professional model. For some specific categories of tasks, the industry model can complete such tasks more accurately. In the embodiment of the present application, by using a multimodal large model and at least one industry model, it is possible to ensure that the execution results of the task are more accurate.

[0020] In one possible implementation, the video cloud platform has learning capabilities and can learn business knowledge and rules related to the task. The present application does not limit the manner in which the video cloud platform obtains the business knowledge and rules. For example, the video cloud platform can obtain text data (the text data can be provided by the user or retrieved by the video cloud platform from other devices), the text data describes the business knowledge and rules related to the task, and converts the text data into structured data. For another example, the video cloud platform directly receives structured data input by the user that describes the business knowledge and rules related to the task.

[0021] Through the above method, the video cloud platform can obtain task-related business knowledge and rules in different ways, effectively expanding the application scenarios.

[0022] In one possible implementation, when the video cloud platform calls a multimodal big model to perform a task based on a video, the multimodal big model learns the business knowledge and rules related to the task through structured data; and calls the multimodal big model to perform the task based on the video and the business knowledge and rules related to the task.

[0023] Through the above method, for some tasks, necessary business knowledge and rules are always involved. The video cloud platform has the ability to learn the business knowledge and rules related to the task to ensure that the task can be accurately executed and the accuracy of the execution results.

[0024] In the second aspect, the embodiment of the present application also provides a video cloud platform, which has the behavioral function of implementing the video cloud platform in the method example of the first aspect above. The beneficial effects can be found in the description of the first aspect and will not be repeated here. The function can be implemented by hardware or by executing the corresponding software through hardware. The hardware or software includes one or more modules corresponding to the above functions. In one possible design, the structure of the video cloud platform includes a video access module, a task understanding module, and a task execution module. These modules can perform the corresponding functions in the method example of the first aspect above. Please refer to the detailed description in the method example for details, which will not be repeated here.

[0025] In a third aspect, an embodiment of the present application further provides a computing device having the function of implementing the behavior of the video cloud platform in the method example of the first aspect above. The beneficial effects can be found in the description of the first aspect and will not be repeated here. The structure of the computing device includes a processor and a memory, and the processor is configured to support the joint device to perform the corresponding functions in the method of the first aspect above. The memory is coupled to the processor, which stores the necessary computer program instructions and data for the joint device. The structure of the communication device also includes a communication interface for communicating with other devices (such as a video acquisition device).

[0026] In a fourth aspect, an embodiment of the present application further provides a computing device cluster, which has the function of implementing the behavior of the video cloud platform in the method example of the first aspect above. The beneficial effects can be found in the description of the first aspect and will not be repeated here. The computing device cluster includes at least one computing device, and the structure of any computing device includes a processor and a memory. The processor in any computing device is configured to support the video cloud platform to perform part or all of the functions of the first aspect and each possible implementation of the first aspect. The memory is coupled to the processor, which stores the necessary computer program instructions and data for the joint device. The structure of the communication device also includes a communication interface for communicating with other devices.

[0027] In a fifth aspect, the present application further provides a computer-readable storage medium, in which instructions are stored, and when the computer-readable storage medium is run on a computer, the computer executes the method in the above-mentioned first aspect and various possible implementations of the first aspect.

[0028] In a sixth aspect, the present application also provides a computer program product comprising instructions, which, when executed on a computer, enables the computer to execute the method in the above-mentioned first aspect and various possible implementations of the first aspect.

[0029] In a seventh aspect, the present application also provides a computer chip, which is connected to a memory, and the chip is used to read and execute a software program stored in the memory to execute the method in the above-mentioned first aspect and various possible implementations of the first aspect. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Figure 1 A schematic diagram of the architecture of a system provided for this application;

[0031] Figure 2 A schematic diagram of the structure of a video cloud platform provided for this application;

[0032] Figure 3 A schematic diagram of a video processing method provided in this application;

[0033] Figure 4 A schematic diagram of a video processing flow provided for this application;

[0034] Figure 5 A schematic diagram of a video viewing interface provided for this application;

[0035] Figures 6A to 6D A schematic diagram of a task setting interface provided for this application;

[0036] Figure 7 A task understanding process intent provided for this application;

[0037] Figure 8 The intention of a task execution process provided for this application;

[0038] Fig. 9 A schematic diagram of the structure of a computing device provided for this application;

[0039] Fig.10 A schematic diagram of the structure of a computing device cluster provided in this application. DETAILED DESCRIPTION

[0040] Before introducing the object processing method, apparatus, and device provided in the embodiments of the present application, several concepts involved in the embodiments of the present application are explained.

[0041] (1) Large multimodal model.

[0042] A multimodal large model refers to a machine learning model that can process information in different modalities (such as images, voice, text, etc.). The input of a multimodal large model is no longer limited to a certain modality, but can input information in different modalities. Correspondingly, the output of a multimodal large model is no longer limited to a fixed modality, and can output corresponding modal information based on the input modality or input information content of the multimodal large model.

[0043] Multimodal large models are often used in scenarios such as visual question answering, image and text generation, and open object detection and segmentation. Visual question answering refers to outputting the answer to a question based on an input image and a question (the question can be in the form of different modalities such as text or voice). The answer can be in the form of an image, text, or voice. Image and text generation refers to processing the input image according to the input question and outputting the processed image. Open object detection and segmentation refers to performing object detection and segmentation on the input image, detecting the objects in the image, and outputting the object box including the object (outputting the object box of the object is object detection) or the outline of the object (outputting the outline of the object is object segmentation); the so-called "open" means that object detection and segmentation are no longer limited to a specific field, and object detection and segmentation can cover a variety of different fields (for example, person recognition, vehicle recognition, animal recognition, etc.).

[0044] The multimodal large model is a machine learning model formed after training based on massive amounts of information in different modalities. It has good generalization and is more "open".

[0045] (2) Small models or industry models.

[0046] In contrast to the "multimodal large model", a machine learning model that targets a specific field or can only complete a specific task is called a small model. In some scenarios, a small model can also be called an industry model. Small models are more targeted, have a higher degree of completion for specific tasks, and have more accurate results. Small models need to be trained with data sets related to the specific task. License plate detection models, pedestrian flow detection models, face detection modules, and water level detection models are more professional models and belong to small models.

[0047] like Figure 1 As shown, a schematic diagram of a system architecture provided by an embodiment of the present application is provided, and the system includes a video cloud platform 100 and at least one video acquisition device 200. The video cloud platform 100 is connected to the at least one video acquisition device 200, and the video cloud platform 100 and the at least one video acquisition device can directly interact.

[0048] For any video acquisition device 200, the video acquisition device 200 can acquire video and transmit the acquired video to the video cloud platform. The present application does not limit the specific type of the video acquisition device 200. The video acquisition device 200 can be a video camera, a monitoring device with a specific video shooting function, a digital camera, a mobile terminal, a laptop computer, a desktop computer with a camera function, a smart home device with a camera function, and other computing devices. Any device capable of video acquisition can be used as the video acquisition device 200.

[0049] For the video cloud platform 100, facing the video acquisition device 200, the video cloud platform 100 can obtain the video acquired by the video acquisition device 200, process and save the acquired video. The present application does not limit the way the video cloud platform 100 processes the video. For example, the video cloud platform 100 can extract some image frames from the acquired video and save them. For another example, the video cloud platform 100 can add scene tags to the acquired video and save it, or add scene tags to the extracted partial image frames and save them. For another example, the video cloud platform 100 can also add information to the acquired video or the extracted partial image frames, and the added information includes but is not limited to: location information representing the location of the video acquisition device 200 that acquires the video or the location where the video is recorded, content description information of the video, time information representing the shooting time of the video, etc.

[0050] For users, the video cloud platform 100 provides an interactive interface. Users can interact with the video cloud platform 100 through the interactive interface. Users can input task description information to the video cloud platform through the interactive interface to set a task for the video cloud platform 100, and the task is the task included in the task description information. After completing the task based on the saved video, the video cloud platform 100 provides the user with the execution result of the task through the interactive interface.

[0051] The interactive interface mentioned in the embodiments of the present application refers to a function for interacting with the user provided to the user, and the embodiments of the present application do not limit the specific presentation method of the interactive interface. For example, the interactive interface can be understood as client software deployed on the user's device or a dedicated, user-oriented client device. For another example, the interactive interface can be understood as a website that the user can access or a web page built based on hypertext markup language (HTML).

[0052] In the embodiment of the present application, the video cloud platform 100 does not set a standard for the expression method of the task description information input by the user through the exchange interface, and the user can input the task description information according to his or her own expression habits. In other words, the user can interact with the video cloud platform 100 through the interactive interface according to his or her own needs and his or her own expression habits, and input the task description information. The video cloud platform 100 itself has the ability to understand the semantics of the task description information input by the user, that is, the video cloud platform 100 supports multimodal understanding of the task description information.

[0053] The task description information input by the user is no longer a fixed one or several specific instructions, but is proposed by the user according to his own needs. In addition, the task description information input by the user is generated according to the user's own expression habits, that is, the structure of the task description information is not fixed, including the task description information of the same task can have a variety of different expressions, and for such a variety of different expressions of the task description information, the video cloud platform and 100 can parse out the task. For example, even if the user expresses the task description information in the form of voice, according to different users' mastery of voice and expression habits, the task description information of the same task can also have different expressions. In the embodiment of the present application, the video cloud platform 100 has this parsing ability and can identify the task from the task description information formed by different expressions. For another example, for the task description information of the same task, there may also be task description information of different modes. In the embodiment of the present application, the video cloud platform 100 can still parse out the task from the task description information of different modes.

[0054] After the user inputs the task description information through the interactive interface, the video cloud platform 100 obtains the task description information, understands (also referred to as parsing) the task description information, parses the intent of the task description information, generates a task corresponding to the task description information, and executes the task based on the saved video. When executing the task, the video cloud platform 100 can also generate one or more subtasks based on the task. The video cloud platform 100 executes the one or more subtasks based on the saved video.

[0055] When performing the task, the video cloud platform 100 can use a machine learning model to complete the task. This application does not limit the number and type of machine learning models used by the video cloud platform 100. The machine learning model used by the video cloud platform 100 can be a large multimodal model. The machine learning model used by the video cloud platform 100 can also include a large multimodal model and one or more small models.

[0056] After executing the task, the video cloud platform 100 obtains the execution result of the task and feeds back the execution result of the task to the user.

[0057] The video cloud platform 100 can also learn business knowledge and rules related to videos. For example, the video managed by the video cloud platform 100 is a video of vehicles at an intersection, and the video cloud platform 100 can learn the movement rules and traffic rules of vehicles at intersections. For another example, the video managed by the video cloud platform 100 is a video within an enterprise park, and the video cloud platform 100 can pre-learn the license plates of the park vehicles registered in the enterprise park and the facial information of the staff members entered in the enterprise park. For another example, the video managed by the video cloud platform 100 is a video within a park, and the video cloud platform 100 can obtain the code of conduct of the park and the opening hours of the park. The business knowledge and rules related to the video are information that the video cloud platform 100 may need to perform the user's tasks, and the video cloud platform 100 can learn the business knowledge and rules in advance.

[0058] In an embodiment of the present application, the video cloud platform 100 can connect to multiple video acquisition devices 200, and can obtain videos collected by the multiple video acquisition devices 200. The user can input task description information to the video cloud platform 100, and the task description information supports multimodality. The video management module analyzes and understands the user's task description information, obtains the task from it, calls the multimodal large model based on the acquired video to perform the task, and feeds back the execution result of the task. The video cloud platform 100 can obtain a large number of videos, no longer limited to a certain video or a certain type of video, and can also complete the tasks set by the user based on the large number of videos obtained. This also makes the acquisition of a large number of videos ensure that the video cloud platform 100 can complete a variety of different types of tasks, which is more intelligent.

[0059] Regarding the form of the video cloud platform 100, the video cloud platform 100 can be implemented by software or hardware. As an example, the implementation of the video cloud platform 100 is described below.

[0060] The video cloud platform 100 is taken as an example of a software functional unit. The video cloud platform 100 may include code running on a computing instance. Among them, the computing instance may be at least one of a physical host (computing device), a virtual machine, a container and other computing devices. Furthermore, the above-mentioned computing device may be one or more. For example, the video cloud platform 100 may include code running on multiple hosts / virtual machines / containers. It should be noted that the multiple hosts / virtual machines / containers used to run the application may be distributed in the same region or in different regions. The multiple hosts / virtual machines / containers used to run the code may be distributed in the same availability zone (AZ) or in different AZs, each AZ including a data center or multiple data centers with similar geographical locations. Among them, usually a region may include multiple AZs.

[0061] Similarly, multiple hosts / virtual machines / containers used to run the code can be distributed in the same virtual private cloud (VPC) or in multiple VPCs. Usually, a VPC is set up in a region. For cross-region communication between two VPCs in the same region and between VPCs in different regions, a communication gateway must be set up in each VPC to achieve interconnection between VPCs through the communication gateway.

[0062] The video cloud platform 100 is taken as an example of a hardware functional unit, and the video cloud platform 100 may include at least one computing device, such as a server, etc. Alternatively, the video cloud platform 100 may also be a device implemented using an application-specific integrated circuit (ASIC) or a programmable logic device (PLD). The PLD may be a complex programmable logical device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL), or any combination thereof.

[0063] The multiple computing devices included in the video cloud platform 100 can be distributed in the same region or in different regions. The multiple computing devices included in the video cloud platform 100 can be distributed in the same AZ or in different AZs. Similarly, the multiple computing devices included in the video cloud platform 100 can be distributed in the same VPC or in multiple VPCs. The multiple computing devices can be any combination of computing devices such as servers, ASICs, PLDs, CPLDs, FPGAs, and GALs.

[0064] The structure of the video cloud platform 100 is described below. Figure 2 As shown, the present application also provides a video cloud platform 100 , which includes a video access module 110 , a task understanding module 120 , and a task execution module 130 .

[0065] The video access module 110 is connected to one or more video acquisition devices 200 , and obtains the video acquired by the one or more video acquisition devices 200 from the one or more video acquisition devices 200 .

[0066] The video access module 110 has a video processing function. After acquiring the video, the video access module 110 can further process the acquired video and save the processed video after processing the video.

[0067] The embodiment of the present application does not limit the processing operation of the video access module 110 on the video. Several processing operations that the video access module 110 can perform are listed below.

[0068] ①. Video frame extraction or short video extraction.

[0069] Video frame extraction refers to the video access module 110 extracting some image frames from the video and discarding the remaining image frames. The present application does not limit the specific method of video frame extraction. For example, the video access module 110 can extract image frames from the video at intervals of fixed duration, such as extracting one image frame every 1 minute. For another example, the video access module 110 can extract image frames from the video at intervals of a fixed number of image frames, such as extracting one image frame every 100 image frames. For another example, the video access module 110 can group the video, and the similarity of the image frames included in each group is greater than the similarity threshold (the calculation method of the similarity of the image frames is not limited here, for example, the similarity between the image frames can be calculated based on the pixel values ​​of each image frame or the targets included), and the video access module 110 can extract one or more image frames from each group of image frames.

[0070] Short video extraction means that the video access module 110 divides the video into multiple short videos of shorter duration, and the video access module 110 only retains some of the short videos among the multiple short videos. The embodiment of the present application does not limit the way in which the video access module 110 divides the video. For example, the video access module 110 can divide the video into multiple short videos of the same duration. For another example, the video access module 110 can divide the video into multiple short videos according to the time of video acquisition, such as the video docking module 110 can divide the video into videos during the morning peak (7 am to 9 am), videos during the evening peak (5 pm to 7 pm), and videos during non-peak hours. The present application also does not limit the way in which the video access module 110 selects some of the short videos to be retained from the multiple short videos after segmentation. For example, the video access module 110 only retains some short videos with a low degree of similarity. For another example, the video access module 110 only retains short videos in important time periods, such as videos during the morning and evening peak hours.

[0071] Video frame extraction and short video extraction can effectively reduce the amount of video data that needs to be kept, avoiding the need to analyze a large amount of video when performing tasks set by the user later.

[0072] ②. Scene label management.

[0073] The video access module 110 can identify the scene in the video, and after identifying the scene in the video, it can add a scene tag to the video. The "scene" here mainly refers to the place or place shown in the video. For example, intersections, schools, squares, railway stations, parks, park gates, venues, etc.

[0074] The setting of scene tags enables the videos acquired by the video access module 110 to be categorized, so that when performing the tasks set by the user later, the video to be analyzed can be accurately acquired from the saved videos according to the scene tags.

[0075] ③. Video content recognition.

[0076] The video access module 110 can perform a preliminary analysis of the content displayed in the video to determine the content displayed in the video, or preliminarily determine the content displayed in the video and add content description information to the video, where the content description information is used to indicate the content displayed in the video. The video access module 110 can determine the content displayed in the video by analyzing the types of objects that appear in the video. For example, when the video access module 110 determines that targets such as intersections and vehicles appear in the video, it can be determined that the video content displays the movement of vehicles at the intersection. For another example, when the video access module 110 determines that targets such as students and school gates appear in the video, it can be determined that the video content displays the flow of people at the school gate.

[0077] Video content recognition adds content description information to the videos saved by the video access module 110, ensuring that when the tasks set by the user are subsequently executed, relevant videos can be accurately obtained from them based on the video content description information, avoiding the need to perform content recognition on the saved videos one by one when executing the tasks set by the user.

[0078] ④. Management of video related information.

[0079] The video access module 110 can also obtain relevant information of the video, such as location information representing the location of the video acquisition device 200 that acquires the video or the location where the video is shot, time information representing the date and time when the video is shot, video data volume, etc. The video access module 110 can associate the acquired relevant information of the video with the video and save it.

[0080] In addition to the above-mentioned processing operations, the video access module 110 can also perform other processing operations, such as playing, exporting to other devices, etc. For example, the video access module 110 can play the saved video or import the video to other devices in response to the user's instructions or operations.

[0081] The video access module 110 also has storage capabilities. A video database 111 may be provided in the video access module 110. The video database 111 may include videos processed by the video access module 110 (eg, videos with added scene tags and content description information) and related information of the videos.

[0082] The task understanding module 120 can interact with the user through an interactive interface, receive tasks set by the user, and feed back the execution results of the tasks to the user through the interactive interface.

[0083] The foregoing description mentions that the video cloud platform 100 does not set a standard for the way of expressing the task description information input by the user to the video cloud platform 100, that is, the task description information input by the user through the interactive interface can adopt different modes. For example, the user uses voice expression to input task description information to the video cloud platform 100 through the interactive interface, and the task description information exists in the form of voice. For another example, the user uses text expression to input task description information to the video cloud platform 100 through the interactive interface, and the task description information exists in the form of text. For another example, the user uses image expression to input task description information to the video cloud platform 100 through the interactive interface, and the task description information exists in the form of. For another example, the user uses sign language or gestures through the interactive interface to input task description information to the video cloud platform 100, and the task description information exists in the form of action.

[0084] The task understanding module 120 can understand the task description information with different modalities, parse the task description information, and determine the task corresponding to the task description information. The task understanding module 120 can also determine the video involved in the task (that is, the target video required to perform the task). The task understanding module 120 can obtain the target video required to perform the task from the scene label, content description information or related information of the video saved by the video access module 110.

[0085] For example, if the task set by the user is "Please help me check whether there is traffic congestion in the XX area between 7:00 and 8:00 a.m.", the task understanding module 120 determines, by analyzing the task, that the video involved in the task is a video captured by a video acquisition device located in the XX area between 7:00 and 8:00 a.m. The task understanding module 120 can find the video in the XX area by searching the scene tags and content description information of the video, and find the video between 7:00 and 8:00 a.m. according to the relevant information of the video.

[0086] The task understanding module 120 can also obtain the business knowledge and rules involved in the video. The embodiment of the present application does not limit the way in which the task understanding module 120 obtains the business knowledge and rules involved in the video. For example, the user can input structured data (such as a data table) indicating the business knowledge and rules to the task understanding module 120 in advance, and the task understanding module 120 can directly recognize the structured data. For another example, the user can send text data representing the business knowledge and rules to the task understanding module 120. The task understanding module 120 can use its own learning ability to parse the text data and convert the text information into structured data that it can use. Among them, structured data refers to data with a neat structure.

[0087] The task understanding module 120 may further decompose the task into multiple subtasks, and deliver the multiple subtasks and the related videos to the task execution module 130 .

[0088] Still taking the task set by the user as “Please help me check whether there is traffic congestion in the XX area between 7 and 8 in the morning” as an example, the task understanding module 120 can convert the task into a subtask of “analyzing whether there is a vehicle in the video” and a subtask of “determining whether there is traffic congestion from a video with a vehicle”. The task understanding module 120 transmits the video involved in the task and the two subtasks to the task execution module 130.

[0089] Of course, the task understanding module 120 may not decompose the task, but further simplify it to generate a subtask. For example, the task understanding module 120 may convert the task into a subtask of "analyzing whether there is traffic congestion in the video image". The task understanding module 120 transmits the video involved in the subtask and the subtask to the task execution module 130.

[0090] The task understanding module 120 can convert the task set by the user into a subtask that can be recognized or processed by the task execution module 130, so that the subsequent task execution module 130 can accurately process the subtask and feedback the execution result of the subtask.

[0091] After transmitting the video involved in the task and the subtask to the task execution module 130 , the task understanding module 120 can also obtain the execution result of the subtask from the task execution module 130 and feed back the execution result of the task to the user through the interactive interface.

[0092] Taking the example of the task understanding module 120 transmitting the video involved in the task and the subtask of “analyzing whether there is traffic congestion in the video footage” to the task execution module 130, the task understanding module 120 can obtain the video footage with traffic congestion from the task execution module 130 or obtain the execution result of “no video footage with traffic congestion found” from the task execution module 130.

[0093] If the task understanding module 120 obtains a video image showing traffic congestion from the task execution module 130, the task understanding module 120 may notify the user of the traffic congestion through the interactive interface, and display the video image showing traffic congestion to the user, as well as relevant information of the video image, such as the location and time of the video acquisition device 200 that acquired the video.

[0094] If the task understanding module 120 obtains the execution result of “no video footage with traffic congestion found” from the task execution module 130 , the task understanding module 120 may inform the user through the interactive interface that there is no traffic congestion.

[0095] The task understanding module 120 can provide the user with feedback on the execution result of the task through different modalities such as voice, text or image.

[0096] The task execution module 130 is the main module for executing the task in the video cloud platform 100. The embodiment of the present application does not limit the way in which the task execution module 130 executes the subtask transmitted by the task understanding module 120. Exemplarily, the task execution module 130 includes a multimodal large model. Optionally, the task execution module 130 may also include one or more small models. The multimodal large model may be connected to the one or more small models. The multimodal large model may transmit the subtasks that need to be processed by the small model to the small model.

[0097] Still taking the subtask "analyzing whether there is traffic congestion in the video screen" received by the task execution module 130 from the task understanding module 120 as an example, the multimodal large model in the task execution module 130 can analyze the video obtained from the task execution module 130 to determine whether there is traffic congestion. For the video with traffic congestion, the license plate can be further identified from it. The multimodal large model can transmit the determination from the video with traffic congestion to the small model for identifying the license plate, and the small model can identify the license plate from the video with traffic congestion.

[0098] Among them, the video access module 110, the task understanding module 120, and the task execution module 130 can all be implemented by software, or can be implemented by hardware. The implementation of the video access module 110, the task understanding module 120, and the task execution module 130 is similar to the implementation of the video cloud platform 100. For details, please refer to the above content and will not be repeated here.

[0099] Combine the following Figure 3 A video processing method provided in an embodiment of the present application is described, and the method includes:

[0100] Step 300: The video cloud platform 100 obtains a video captured by at least one video capture device 200. This step may be performed by the video access module 110.

[0101] The video cloud platform 100 is connected to at least one video acquisition device 200, and the video cloud platform 100 and the at least one video acquisition device 200 can interact through the connection between them. The embodiment of the present application does not limit the way in which the video cloud platform 100 obtains the video captured by at least one video acquisition device 200.

[0102] The video cloud platform 100 may actively acquire video from the at least one video acquisition device 200. For example, the video cloud platform 100 may periodically acquire video from the at least one video acquisition device 200. For another example, the video cloud platform 100 may acquire video from the at least one video acquisition device 200 according to a user's instruction (such as an instruction to acquire a video or an instruction to save a video).

[0103] The at least one video acquisition device 200 may also actively send the acquired video to the video cloud platform 100. For example, a video upload time is set for the at least one video acquisition device 200, and when the time reaches the set video upload time, the at least one video acquisition device 200 may send the acquired video to the video cloud platform 100. For another example, the at least one video acquisition device 200 may store the acquired video in a storage device in the cloud, and notify the video cloud platform 100 that the video has been uploaded, and the video cloud platform 100 may obtain the video from the storage device in the cloud after receiving the notification from the at least one video acquisition device 200. For another example, the at least one video acquisition device 200 may store the acquired video in a storage device in the cloud, and the video cloud platform 100 may periodically obtain the video from the storage device in the cloud.

[0104] In the embodiment of the present application, the video cloud platform 100 is not limited to interacting with one video acquisition device 200, and the video cloud platform 100 can interact with multiple video acquisition devices 200 to obtain videos acquired by the multiple video acquisition devices 200. This method is suitable for scenes involving multiple video acquisition devices 200, such as enterprise parks, parks, urban transportation, buildings, residential areas, and other public places.

[0105] Taking an enterprise park as an example, the video cloud platform 100 can interact with multiple cameras (ie, the video acquisition device 200) installed in the enterprise park, obtain the videos shot by the cameras of the enterprise park, and manage them in a unified manner.

[0106] Step 301: The video cloud platform 100 processes the acquired video and saves the processed video. This step can be performed by the video access module 110.

[0107] The video cloud platform 100 can directly save the captured video after acquiring the video captured by the at least one video capture device 200. The video cloud platform 100 can also process the video and save the processed video. The video cloud platform 100 processes the video including but not limited to: video frame extraction, short video extraction, scene tag management, video content video and video related information management, and the video cloud platform 100 can perform some or all of the above-listed processing operations on the video, and can also perform operations other than the above-listed processing operations.

[0108] like Figure 4 As shown, the video access module 110 in the video cloud platform 100 acquires the video and processes the video. The video access module 110 can acquire the video from multiple video acquisition devices 200, and after acquiring the video, uniformly process the video. The video access module 110 can perform scene tag management, set scene tags for the video, and can also perform video frame extraction for the acquired video to obtain the position information representing the position of the video acquisition device 200 that shot the video or the position where the video was shot, as well as the time information representing the shooting time of the video. The processed video is then stored in the video database 111, so that the video in the video database 111 includes the video's position information, scene tags, content description information, and time information.

[0109] Still taking the enterprise park as an example, after obtaining the video shot by the camera in the enterprise park, the video cloud platform 100 can parse the obtained video to generate scene tags and content description information for the video, and can also perform operations such as video frame extraction or short video extraction, and save the processed video.

[0110] In addition, the video cloud platform 100 can also retain some unprocessed videos (such as retaining videos within a specific time period) so that users can view the videos later.

[0111] The video cloud platform 100 can provide users with video viewing and management functions through an interactive interface. Figure 5 The figure shows a schematic diagram of the viewing and management interface provided by the video cloud platform 100 to the user through the interactive interface.

[0112] exist Figure 5 In the diagram, the video cloud platform 100 can display a list of videos within a specific time period (such as the current month) to the user. In the viewing and management interface diagram, the video cloud platform 100 can display a list of videos to the user according to date and location. Figure 5 The video list collected in July 2023 and the video list of different locations in July 2023 are shown in an exemplary manner. The user selects a video in the video list, such as by touching the screen, operating the mouse or keyboard or other input and output devices, or by voice, to view the video content information (such as traffic or pedestrian flow), scene labels (such as building 1, building 2, leisure area 1, leisure area 2, etc.) of the video, and can also view the video and browse the content of the video.

[0113] The video cloud platform 100 can obtain and process the video in advance through steps 300 to 301 to facilitate the subsequent execution of tasks set by the user.

[0114] Step 302: The user inputs task description information to the video cloud platform 100 through the interactive interface, and the task description information includes some or all of the following modes: voice, text, image, and action. The operation performed in the video cloud platform 100 in this step can be performed by the task understanding module 120, that is, the task understanding module 120 can accept the task set by the user through the interactive interface.

[0115] The user can input task description information to the video cloud platform 100 according to his / her own needs, and the video cloud platform 100 does not limit the specific mode of the task description information set by the user. In other words, the video cloud platform 100 can parse task description information of different modes.

[0116] like Fig. 6A FIG. 1 is a schematic diagram of a task configuration interface provided to a user by the video cloud platform 100 through the interactive interface. The video cloud platform 100 can prompt the user to set a task.

[0117] The user can directly input the task description information by voice. The video cloud platform 100 detects the user's voice and recognizes the voice. It can display the content of the recognized voice in text in the task configuration interface. After the user clicks to confirm, the user can be notified that the task has been successfully received. The video cloud platform 100 detects the user's voice and recognizes the voice. It can also repeat the task set by the user by voice and ask the user whether it is correct. After detecting affirmative information such as "confirm", "agree" or "yes" expressed by the user through voice, the user is informed by voice that the task has been successfully received.

[0118] For example, the user can use voice to input the task description information ""According to the company's vehicle management regulations, please help me check whether there are any vehicles that illegally enter between 7 am and 8 am." Fig. 6A In the task setting interface, the video cloud platform 100 can prompt the user whether the task entered by the user is "According to the company's vehicle management regulations, please help me check whether there are any vehicles that enter illegally between 7 am and 8 am". When the user expresses "yes" in voice, the video cloud platform 100 can prompt the user in the task setting interface that "the task has been received successfully".

[0119] The user can also input the task description information through text. The user directly types the text in the task setting interface and completes the task setting by clicking "Confirm". The video cloud platform 100 notifies the user that the task has been successfully received by detecting the text typed by the user and clicking "Confirm".

[0120] For example, the user can enter the task description information in text: "According to the company's vehicle management regulations, please help me check whether there are any vehicles that illegally enter between 7 am and 8 am", and click "Confirm" after entering the text. Figure 6B In the task setting interface, the video cloud platform 100 can prompt the user that "task received successfully" in the task setting interface.

[0121] The above lists only two modes of tasks. In fact, the tasks set by the user through the interactive interface can have multiple modes at the same time.

[0122] For example, the user can input the task description information in a combination of text and image. The user types text and uploads an image in the task setting interface, and completes the task setting by clicking "Confirm". The video cloud platform 100 notifies the user that the task has been successfully received by detecting the text typed by the user and the operation of clicking "Confirm".

[0123] For example, the user can enter the task description information "Please help me check whether there is a vehicle in the image entering between 7 am and 8 am" in text, upload the image of the vehicle, and click "Confirm" after entering the text and uploading the image. Figure 6C In the task setting interface, the video cloud platform 100 can prompt the user that "task received successfully" in the task setting interface.

[0124] For example, the user can input the task description information through a combination of voice and image. The user uploads an image in the task setting interface and inputs the rest of the task description information through voice. After detecting the user's voice and the uploaded image, the video cloud platform 100 recognizes the voice and image, and can display the content of the recognized voice and image in text in the task configuration interface. After the user clicks to confirm, the user can be notified that the task has been successfully received. The video cloud platform 100 detects the user's voice, recognizes the voice, and can also repeat the task description information entered by the user through voice, asking the user whether it is correct. After detecting affirmative information such as "confirm", "agree" or "yes" expressed by the user through voice, the user is informed through voice that the task has been successfully received.

[0125] For example, the user can use voice to input the task description information "Please help me check whether there is a vehicle in the image entering between 7 am and 8 am". Fig.6D In the task setting interface, the video cloud platform 100 can prompt the user whether the task entered is "Please help me check whether there is a vehicle in the image entering between 7 am and 8 am". When the user expresses "yes" in voice, the video cloud platform 100 can prompt the user in the task setting interface that "the task has been received successfully".

[0126] Step 303: The video cloud platform 100 analyzes the set task, executes the task based on the saved video, and obtains the execution result of the task. In this step, the task analysis operation can be performed by the task understanding module 120, and the task execution operation in this step can be performed by the task execution module 130.

[0127] The following describes how the task understanding module 120 and the task execution module 130 cooperate to perform this step:

[0128] Step 3031: The task understanding module 120 parses the task description information and generates a task corresponding to the task description information. The task understanding module 120 may generate one or more subtasks based on the task.

[0129] Since the task description information input by the user through the interactive interface is set by the user's expression habits, that is, closer to human natural language, the task understanding module 120 needs to further understand the task description information and parse it into a machine language that the task execution module 130 can understand.

[0130] Taking the task description information input by the user as "According to the company's vehicle management regulations, please help me check whether there are any vehicles that enter illegally between 7 and 8 in the morning" as an example, the task understanding module 120 can understand the task description information, and its corresponding task is "Find non-company vehicles that enter the enterprise park between 7 and 8 in the morning."

[0131] The task understanding module 120 can recognize the user's natural language, understand the intention of the task description information, and generate a task corresponding to the task description information, so that the subsequent task execution module 130 can accurately execute the task.

[0132] The task understanding module 120 may also split the parsed task into multiple subtasks so that the task execution module 130 can execute the multiple subtasks.

[0133] Among them, splitting a task into multiple subtasks can split a relatively complex task into multiple simpler subtasks, so that the task execution module 130 can execute the task more efficiently. In addition, for the case of multiple independent subtasks, that is, the execution of one subtask will not affect the execution of another subtask, so that multiple subtasks can be executed in parallel, the execution efficiency of the task can be more effectively improved.

[0134] Of course, for some simple tasks or when the performance of the task execution module 130 is relatively strong, the task understanding module 120 may not split the task, but directly transmit the understood task to the task execution module 130. For the convenience of expression, in the embodiment of the present application, "generating a subtask based on the task" is used to indicate that the task is not split.

[0135] Step 3032: The task understanding module 120 searches for the target video required to perform the task from the saved videos.

[0136] After understanding the task, the task understanding module 120 can find the target video required to execute the task from the videos saved by the video access module 110 for the task execution module 130 to execute the task.

[0137] Still taking the task of “finding non-company vehicles entering the enterprise park between 7:00 and 8:00 a.m.” as an example, the task understanding module 120 can query the target video of the enterprise park entrance and exit between 7:00 and 8:00 a.m. from the videos saved by the video access module 110 .

[0138] The task understanding module 120 can also retrieve the license plates of the company's registered vehicles and convert the license plates of the company's registered vehicles into structured data that can be recognized by the task execution module 130. For example, the task understanding module 120 retrieves the company's license plate database data A = {aaa, bbb, ...}.

[0139] In this way, the task can be further converted into "Is there a vehicle entering the video? If so, is its license plate number in the following set: A = {aaa, bbb, ...}".

[0140] This task can be divided into two subtasks: subtask 1 is “detecting whether there is a vehicle entering the video”, and subtask 2 is “detecting whether the license plate in the video with a vehicle entering is in the following set: A = {aaa, bbb, …}”.

[0141] like Figure 7 As shown, the task understanding module 120 executes the process of step 3031 and step 3032. The task understanding module 120 can retrieve relevant videos and the company's license plate database data.

[0142] The embodiment of the present application does not particularly emphasize the execution order of step 3031 and step 3032, and the task understanding module 120 can execute the two steps synchronously. Here, the task understanding module 120 is divided into two steps for the sake of ease of understanding, so as to show the process of the task understanding module 120 executing the operation.

[0143] Step 3033 : The task understanding module 120 transmits the one or more subtasks and the video involved in the task to the task execution module 130 .

[0144] After completing the task understanding operation, the task understanding module 120 may transmit the acquired information to the task execution module 130. In the scenario where the task is "find non-company vehicles entering the enterprise park between 7 am and 8 am", the task understanding module 120 may transmit subtask 1 and subtask 2, as well as the video of the entrance and exit of the enterprise park between 7 am and 8 am to the task execution module 130.

[0145] Step 3034 : the task execution module 130 executes the one or more subtasks, and feeds back the execution results of the one or more subtasks to the task understanding module 120 .

[0146] The task execution module 130 may call the multimodal large model to execute the one or more subtasks, or may call the multimodal large model and the small model to execute the one or more subtasks.

[0147] In the process of executing subtask 1 and subtask 2 by task execution module 130, task execution module 130 can call the multimodal large model to execute subtask 1 and find the video with vehicle entry and exit. The multimodal large model outputs the target frame of the vehicle in the video or the outline of the vehicle through open target detection and segmentation. The output of the multimodal large model can be in the form of a rendered image or structured data that records the coordinates of the rectangular frame and the segmented outline.

[0148] For videos with vehicles entering or exiting, the task execution module 130 can call the multimodal large model and the small model for subtask 2. The multimodal large model can transmit the target frame of the vehicle or the outline of the vehicle to the small model for license plate recognition. After receiving the video with vehicles entering or exiting, the small model for license plate recognition can identify the license plate of the vehicle appearing in the video, and transmit the identified license plate to the multimodal large model. The multimodal large model then determines whether the license plate belongs to the set A = {aaa, bbb, ...} based on the license plate of the vehicle received from the small model, and finds the license plate that does not belong to the set A = {aaa, bbb, ...}.

[0149] like Figure 8 As shown in FIG. 1 , the task execution module 130 executes the task. The task execution module 130 obtains the split subtask 1 and subtask 2, as well as the video involved in the task, from the task understanding module 120. The multimodal large model can execute subtask 1 and subtask 2, and as early as when executing subtask 3, the small model can be called to perform license plate recognition.

[0150] In the above example, subtask 1 and subtask 2 need to be executed serially, that is, the execution of subtask 2 depends on the execution result of the subtask. In practical applications, multiple subtasks split by the task understanding module 120 may also be executed in parallel. For example, the task set by the user is "find non-company vehicles and non-company personnel entering the enterprise park between 7 and 8 in the morning". After the task understanding module 120, the task can be split into the following four tasks: subtask 1 is "detect whether there is a vehicle entering in the video", subtask 2 is "whether the license plate of the vehicle entering is in the following set: A = {aaa, bbb, ...}", subtask 3 is "detect whether there is a person entering in the video", and subtask 4 is "whether the face of the person in the video entering is in the following set: B = {aaa, bbb, ...}", and B = {aaa, bbb, ...} is the facial feature set of company personnel.

[0151] Subtask 1 and subtask 3 can be executed in parallel, and subtask 2 and subtask 4 can be executed in parallel. That is, the task execution module 130 can call the multimodal large model to execute subtask 1 and subtask 3 in parallel, and the multimodal large model can also transmit the execution results of subtask 1 and subtask 3 in parallel to the submodel for license plate recognition and the submodel for face recognition, respectively, and detect the license plates of vehicles other than the company and non-company personnel according to the outputs of the submodel for license plate recognition and the submodel for face recognition.

[0152] It is worth noting that when the task execution module 130 executes one or more subtasks, the features mentioned in the video, such as the features of vehicles or personnel extracted from the video, can be reused. That is, when executing each subtask, the features of the extracted video can be reused, thereby reducing the workload and time consumption of the task execution module 130 in executing the task and improving the execution efficiency of the task.

[0153] Step 3034 : the task execution module 130 obtains the execution result of the task based on the execution results of the one or more subtasks, and feeds it back to the task understanding module 120 .

[0154] After executing the one or more subtasks, the task execution module 130 can obtain the execution result of the task. Taking the task understood by the task understanding module 120 as "finding non-company vehicles entering the enterprise park between 7 and 8 a.m." as an example, since the task understanding module 120 has split the task into subtask 1 and subtask 2 that need to be executed serially, the task execution module 130 can use the execution result of subtask 2 as the execution result of the task, and the task execution module 130 can also feed back the execution result of the task to the task understanding module 120.

[0155] For example, the execution result of subtask 2 can be expressed as follows:

[0156] “Video screen A, time 7:10, license plate number XXXXX”;

[0157] "Video screen B, time 7:39, license plate number XXXXX".

[0158] Taking the task understood by the task understanding module 120 as "finding non-company vehicles and non-company personnel entering the enterprise park between 7 and 8 a.m." as an example, since the task understanding module 120 has split the task into subtask 1 and subtask 2 that need to be executed serially, and subtask 3 and subtask 4 that need to be executed serially, the task execution module 130 can use the execution results of subtask 2 and the execution results of subtask 4 as the execution results of the task, and the task execution module 130 can also feed back the execution results of the task to the task understanding module 120.

[0159] For example, the execution results of subtask 2 and subtask 4 can be expressed as follows:

[0160] “Video screen A, time 7:10, license plate number XXXXX”;

[0161] “Video screen B, time 7:39, license plate number XXXXX”;

[0162] “Video screen C, time 7:20, face outline C1”;

[0163] "Video screen D, time 7:25, face outline D1".

[0164] Step 304: The video cloud platform 100 feeds back the execution result of the task to the user through the interactive interface. This step can be performed by the task understanding module 120.

[0165] After executing the task and obtaining the execution result of the task, the video cloud platform 100 feeds back the execution result of the task to the user. The embodiment of the present application does not limit the way in which the video cloud platform 100 feeds back the execution result of the task to the user. The video cloud platform 100 can feed back the execution result of the task to the user in a multi-modal presentation form such as text, image, and voice.

[0166] The present application also provides a computing device 900. Fig. 9 As shown, the computing device 900 includes: a bus 901, a processor 902, a memory 903 and a communication interface 904. The processor 902, the memory 903 and the communication interface 904 communicate with each other through the bus 901. The computing device 900 can be a server or a terminal device. It should be understood that the present application does not limit the number of processors and memories in the computing device 900.

[0167] The bus 901 may be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus. The bus may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Fig. 9 The bus 901 is represented by only one line, but it does not mean that there is only one bus or one type of bus. The bus 901 may include a path for transmitting information between various components of the computing device 900 (for example, the memory 903, the processor 902, and the communication interface 904).

[0168] The processor 902 may include any one or more of a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor (MP), or a digital signal processor (DSP).

[0169] The memory 903 may include a volatile memory, such as a random access memory (RAM). The processor 902 may also include a non-volatile memory, such as a read-only memory (ROM), a flash memory, a hard disk drive (HDD), or a solid state drive (SSD).

[0170] The memory 903 stores executable program codes, and the processor 902 executes the executable program codes to respectively implement the functions of the aforementioned video access module 110, the task understanding module 120, and the task execution module 130, thereby implementing the video processing method. That is, the memory 903 stores computer program instructions for executing the video processing method.

[0171] Alternatively, the memory 903 stores executable codes, and the processor 902 executes the executable codes to implement the functions of the aforementioned video cloud platform 100, thereby implementing the video processing method. That is, the memory 903 stores computer program instructions for executing the video processing method.

[0172] The communication interface 904 uses a transceiver module such as, but not limited to, a network interface card or a transceiver to implement communication between the computing device 900 and other devices or a communication network.

[0173] The embodiment of the present application also provides a computing device cluster. The computing device cluster includes at least one computing device. The computing device can be a server, such as a central server, an edge server, or a local server in a local data center. In some embodiments, the computing device can also be a terminal device such as a desktop computer, a laptop computer, or a smart phone.

[0174] like Fig.10 As shown, the computing device cluster includes at least one computing device 900. The memory 903 in one or more computing devices 900 in the computing device cluster may store the same instructions for executing the application deployment method.

[0175] In some possible implementations, the memory 903 of one or more computing devices 900 in the computing device cluster may also store partial instructions for executing the application deployment method. In other words, the combination of one or more computing devices 900 may jointly execute the video processing method.

[0176] It should be noted that the memory 903 in different computing devices 900 in the computing device cluster can store different instructions, which are respectively used to execute part of the functions of the video cloud platform 100. That is, the instructions stored in the memory 903 in different computing devices 900 can implement the functions of one or more modules in the video access module 110, the task understanding module 120 and the task execution module 130.

[0177] In some possible implementations, one or more computing devices in the computing device cluster may be connected via a network, which may be a wide area network or a local area network.

[0178] The embodiment of the present application also provides a computer program product including instructions. The computer program product may be software or a program product including instructions that can be run on a computing device or stored in any available medium. When the computer program product is run on at least one computing device, the at least one computing device executes an application deployment method or an application deployment method.

[0179] The embodiment of the present application also provides a computer-readable storage medium. The computer-readable storage medium can be any available medium that can be stored by a computing device or a data storage device such as a data center containing one or more available media. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state hard disk). The computer-readable storage medium includes instructions that instruct a computing device to execute an application deployment method, or instructs a computing device to execute an application deployment method.

[0180] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the protection scope of the technical solutions of the embodiments of the present invention.

Claims

1. A video processing method, characterized in that: The method is applied to a video cloud platform connected to multiple video acquisition devices, wherein a multimodal large model is built in the video cloud platform, and the multimodal large model is a machine learning model that supports information input of multiple modalities. The method comprises: Acquire the videos captured by the multiple video capture devices, and save the videos; Providing an interactive interface, wherein the interactive interface allows a user to interact with the video cloud platform; Receiving, through the interactive interface, task description information input by the user, the task description information including some or all of the following modalities: text, voice, image, action; Analyze the intent of the task description information and generate a task corresponding to the task description information; The multimodal large model is called based on the video to execute the task, and the execution result of the task is fed back to the user.

2. The method according to claim 1, characterized in that The calling the multimodal large model based on the video to perform the task includes: Querying the target video required to perform the task from the saved videos; Based on the target video, the task is performed.

3. The method according to claim 1 or 2, characterized in that The step of saving the video comprises: Processing the video, the processing including some or all of the following: adding content description information to the video, setting scene tags, setting location information and time information for the video, video frame extraction, and short video extraction, wherein the location information is used to describe the location where the video was shot, and the time information is used to describe the shooting time of the video; The processed video is saved.

4. The method according to claim 3, characterized in that The step of searching the saved videos for a target video required to perform the task includes: The target video is searched from the video according to part or all of the content description information, scene tag, location information and time information of the video.

5. The method according to claim 1 or 2, characterized in that: The calling the multimodal large model based on the video to perform the task includes: Decomposing the task into a plurality of subtasks; Based on the video, a multimodal large model is called to perform the multiple subtasks.

6. The method according to any one of claims 1 to 5, characterized in that: The multimodal big model is connected to at least one industry model, and the multimodal big model is called based on the video to perform the task, including: The multimodal big model and the at least one industry model are called to perform the task or the subtask.

7. The method according to any one of claims 1 to 6, characterized in that: The method further comprises: Acquire text data, the text data describing business knowledge and rules related to the task, and convert the text data into structured data; or Receive structured data input by a user that describes business knowledge and rules related to the task.

8. The method according to claim 7, characterized in that The calling the multimodal large model based on the video to perform the task includes: Calling the multimodal large model to learn business knowledge and rules related to the task through the structured data; The multimodal large model is called to perform the task based on the video and business knowledge and rules related to the task.

9. A video cloud platform, characterized in that: The video cloud platform is connected to a plurality of video acquisition devices, and the video cloud platform includes: A video access module, used to obtain the videos collected by the multiple video collection devices and save the videos; The task understanding module is used to provide an interactive interface for the user, wherein the interactive interface allows the user to interact with the video cloud platform; receive the task description information input by the user through the interactive interface, wherein the task description information includes some or all of the following modalities: text, voice, image, and action; analyze the intention of the task description information and generate a task corresponding to the task description information; The task execution module includes a multimodal large model, which is a machine learning model that supports information input in multiple modes. The task execution module is used to call the multimodal large model based on the video to execute the task and feedback the execution result of the task to the user.

10. The video cloud platform according to claim 9, characterized in that: The task understanding module is further used to query the target video required to perform the task from the saved videos; The task execution module is used to execute the task based on the target video.

11. The video cloud platform according to claim 9 or 10, characterized in that: When saving the video, the task access module is used to: Processing the video, the processing including some or all of the following: adding content description information to the video, setting scene tags, setting location information and time information for the video, video frame extraction, and short video extraction, wherein the location information is used to describe the location where the video was shot, and the time information is used to describe the shooting time of the video; The processed video is saved.

12. The video cloud platform according to claim 11, characterized in that: The task understanding module searches for a target video required to perform the task from the saved video, for: The target video is searched from the video according to part or all of the content description information, scene tag, location information and time information of the video.

13. The video cloud platform according to claim 9 or 10, characterized in that: The task understanding module is further used to: decompose the task into multiple subtasks; The task execution module is used to: based on the video, call the multimodal large model to execute the multiple subtasks.

14. The video cloud platform according to any one of claims 9 to 13, characterized in that: The multimodal large model is connected to at least one industry model, and the task execution module is used to: The multimodal big model and the at least one industry model are called to perform the task or the subtask.

15. The video cloud platform according to any one of claims 9 to 14, characterized in that: The task understanding module is further used to: Acquire text data, the text data describing business knowledge and rules related to the task, and convert the text data into structured data; or Receive structured data input by a user that describes business knowledge and rules related to the task.

16. The video cloud platform according to claim 15, characterized in that: The task execution module is used to: Calling the multimodal large model to learn business knowledge and rules related to the task through the structured data; The multimodal large model is called to perform the task based on the video and business knowledge and rules related to the task.

17. A computing device cluster, characterized in that: comprising at least one computing device, each computing device comprising a processor and a memory; The processor of the at least one computing device is configured to execute instructions stored in the memory of the at least one computing device, so that the computing device cluster executes the method according to any one of claims 1 to 8.

18. A computer program product comprising instructions, characterized in that When the instructions are executed by a computing device cluster, the computing device cluster executes the method according to any one of claims 1 to 8.

19. A computer-readable storage medium, characterized in that: The method comprises computer program instructions. When the computer program instructions are executed by a computing device cluster, the computing device cluster executes the method according to any one of claims 1 to 8.

Images