Video stream detection method and device, equipment, storage medium and program product

Abstract

The invention discloses a video stream detection method and device, equipment, a storage medium and a program product, and belongs to the technical field of video stream processing. The method comprises the following steps: determining a frame extraction frequency according to an event detection task of a target video stream; according to the frame extraction frequency, determining one of a graphics processing unit (GPU) and a central processing unit (CPU) as a first processor; performing frame extraction and decoding processing on the target video stream through the first processor to obtain a video decoding stream; and executing an event detection task on the video decoding stream to obtain an event detection result. Through the mode, excessive occupation of a single processor resource can be avoided, and processor load balancing is realized, so that the processing efficiency of a video stream detection task is improved.

Description

Technical Field

[0001] The present application relates to the technical field of video stream processing, and in particular to a video stream detection method, device, equipment, storage medium and program product. Background Art

[0002] Video stream encoding and decoding and frame extraction technology is the core technology in the field of video processing, which mainly includes encoding and decoding of video streams and extracting specific frames from them for subsequent processing or analysis. This technology is widely used in many fields such as video surveillance, content review, media playback and video conferencing. At present, video stream encoding and decoding and frame extraction solutions are mainly divided into two categories: one is the frame extraction solution based on the Central Processing Unit (CPU), and the other is the frame extraction solution based on the Graphics Processing Unit (GPU).

[0003] Existing solutions usually rely on a single hardware resource (such as CPU or GPU) to perform video frame extraction and event detection tasks without fully considering the system resource occupancy and remaining computing power, resulting in excessive occupation of hardware resources, thereby affecting the processing efficiency of video stream detection tasks. Summary of the invention

[0004] The embodiments of the present application provide a video stream detection method, apparatus, device, storage medium and program product to at least solve the problem of low processing efficiency of video stream detection tasks in related technologies.

[0005] In order to solve the above technical problems, this application is implemented as follows: In a first aspect, an embodiment of the present application provides a video stream detection method, comprising: determining a frame extraction frequency according to an event detection task of a target video stream; determining, according to the frame extraction frequency, one of a graphics processing unit GPU and a central processing unit CPU as a first processor; performing frame extraction and decoding processing on the target video stream by the first processor to obtain a video decoding stream; executing the event detection task on the video decoding stream to obtain an event detection result.

[0006] In the second aspect, an embodiment of the present application provides a video stream detection device, including: a frame extraction frequency determination module, used to determine the frame extraction frequency according to the event detection task of the target video stream; a processor determination module, used to determine one of a graphics processing unit GPU and a central processing unit CPU as a first processor according to the frame extraction frequency; a frame extraction decoding processing module, used to perform frame extraction decoding processing on the target video stream through the first processor to obtain a video decoding stream; a detection result determination module, used to execute the event detection task on the video decoding stream to obtain an event detection result.

[0007] In a third aspect, an embodiment of the present application provides an electronic device, comprising a processor and a memory, wherein the memory stores programs or instructions that can be run on the processor, and when the programs or instructions are executed by the processor, the steps of the method described in the first aspect above are implemented.

[0008] In a fourth aspect, an embodiment of the present application provides a readable storage medium, on which a program or instruction is stored. When the program or instruction is executed by a processor, the steps of the method described in the first aspect above are implemented.

[0009] In a fifth aspect, an embodiment of the present application provides a computer program product, which includes a computer program stored on a non-transitory computer-readable storage medium, and the computer program includes program instructions. When the program instructions are executed by a computer, the computer executes the steps of the method described in the first aspect above.

[0010] In the embodiment of the present application, the frame extraction frequency is determined according to the event detection task of the target video stream; according to the frame extraction frequency, one of the graphics processor GPU and the central processing unit CPU is determined as the first processor; the first processor performs frame extraction and decoding processing on the target video stream to obtain a video decoding stream; the event detection task is performed on the video decoding stream to obtain an event detection result. In this way, according to the demand of the event detection task for the frame extraction frequency, a suitable processor is selected to perform frame extraction and decoding on the target video stream, which can avoid excessive occupation of a single processor resource, realize processor load balancing, and thus improve the processing efficiency of the video stream detection task.

[0011] It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0013] Figure 1 A schematic diagram of a process flow of a video stream detection method provided in an embodiment of the present application is shown; Figure 2 A schematic diagram of a process flow of an event detection method provided in an embodiment of the present application is shown; Figure 3 Another schematic diagram of a video stream detection method provided in an embodiment of the present application is shown; Figure 4 Shows Figure 3 Schematic diagram of the processing flow of the codec load balancing module; Figure 5A schematic diagram of the structure of a video stream detection device provided in an embodiment of the present application is shown; Figure 6 A schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present application is shown. DETAILED DESCRIPTION

[0014] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present application. Instead, they are merely examples of devices and methods consistent with some aspects of the present application as detailed in the appended claims.

[0015] Currently, video stream encoding, decoding and frame extraction solutions are mainly divided into two types: CPU and GPU. The CPU frame extraction solution usually uses the Fast Forward Moving Picture Experts Group (FFmpeg) tool to extract frames from the video stream and save them as images. The artificial intelligence (AI) algorithm then reads the images, performs preprocessing, reasoning and post-processing, and finally uploads the results. Compared with the CPU solution, the GPU frame extraction solution can greatly improve the encoding and decoding speed in the application of AI algorithms and complete the task more efficiently.

[0016] The relevant solution proposes that the encoding and decoding of video stream data, video processing and AI reasoning tasks are all completed in the GPU, so that the video stream does not flow out of the GPU, making the function of the entire video stream reasoning system more complete. However, a single hardware resource (such as CPU or GPU) may encounter computing bottlenecks in the process of processing the entire video stream encoding and decoding and AI reasoning, resulting in excessive resource occupation, which in turn affects the processing efficiency of the video stream detection task.

[0017] In response to the problems existing in the above-mentioned video stream detection process, an embodiment of the present application provides a video stream detection method. This method selects a suitable processor to perform frame extraction and decoding on the target video stream according to the frame extraction frequency requirements of the event detection task, so as to avoid excessive occupation of a single processor resources, achieve processor load balancing, and thus improve the processing efficiency of the video stream detection task.

[0018] Figure 1The flowchart of the video stream detection method provided in the embodiment of the present application is shown. The execution subject of the method can be a terminal device or a server, wherein the terminal device can be a device such as a personal computer, or a mobile terminal device such as a mobile phone or a tablet computer, and the terminal device can be a terminal device used by a user. The server can be an independent server, or a server cluster composed of multiple servers, and the server can be a background server of a certain business, or a background server of a certain platform or application (such as a security monitoring system, an intelligent transportation system, a video analysis platform, a smart city platform, etc.). In the embodiment of the present application, the execution subject is taken as an example to illustrate. For the case of the terminal device, it can be processed according to the following related content, which will not be repeated here. As shown in the figure, the method 100 may include the following steps: Step 101: Determine the frame extraction frequency according to the event detection task of the target video stream.

[0019] In a specific implementation, it is possible to obtain a target video stream of a preset area collected in real time by a monitoring device such as a camera, or to obtain a pre-stored target video stream to be detected, and set an event detection task for the target video stream, for example, detecting fights, fire detection events, traffic accidents, traffic flow, etc. in the target video stream, wherein the same target video stream can correspond to multiple event detection tasks.

[0020] The frame extraction frequency is determined according to the event detection task. For example, to detect fighting events in the target video, assuming that the frame rate of the target video stream is 30 frames per second (fps), the frame extraction frequency can be set to extract 10 frames per second or higher, which can ensure that the key actions of the fight are captured in time.

[0021] Step 102: According to the frame extraction frequency, determine one of a graphics processor GPU and a central processing unit CPU as a first processor.

[0022] In the specific implementation, the GPU is much more efficient than the CPU in performing frame extraction and decoding of a single video stream, but the video memory resources occupied by the GPU frame extraction and decoding do not depend on the frame extraction frequency, but are only related to the resolution of the video stream and other properties of the video stream; the CPU frame extraction and decoding occupy resources only during the current call, and the resources will be released after the processing is completed. The CPU uses ffmpeg to extract and decode the video stream, and makes interval calls at the frequency required by the task, and releases resources after extracting a picture after each call. Therefore, the GPU is more suitable for tasks that require a higher frequency of video stream frame extraction, and the CPU is suitable for tasks that require a lower frequency of video stream frame extraction. According to the frame extraction frequency corresponding to the event detection task, one of the GPU and the CPU can be determined as the first processor. For example, when the frame extraction frequency f>a, use the GPU for processing; when the frame extraction frequency f≤a, use the GPU for processing, and a can be set according to the resources of the processor.

[0023] Step 103: The target video stream is subjected to frame extraction and decoding processing by the first processor to obtain a decoded video stream.

[0024] In a specific implementation, the target video stream is subjected to frame extraction processing by a selected first processor (such as a GPU), and each extracted frame is decoded. The decoded frames are converted into standard image data to obtain a video decoding stream for subsequent event detection tasks. For example, in a fight scene, the GPU will decode the extracted frames into image data and provide input for subsequent target detection algorithms (such as action recognition, event detection, etc.).

[0025] Step 104: Execute the event detection task on the video decoding stream to obtain an event detection result.

[0026] In a specific implementation, the video decoding stream can be input into a machine learning model, and the machine learning model can be used to detect the targets and behaviors in the video decoding stream to obtain event detection results. Among them, machine learning models include multimodal large models, target detection (YOLO, Faster R-CNN, etc.), action recognition (models based on convolutional neural networks CNN) or time series analysis models (such as long short-term memory LSTM networks).

[0027] Here, a multimodal large model is a machine learning model that can understand and process multiple types of data (i.e., "modalities"), and the types are also called modalities, including text modalities, image modalities, audio modalities, video modalities, etc. This model can integrate information from multiple different modalities and extract useful information from them to perform more complex and intelligent tasks; such as visual question answering (AI interviewer), image and text generation, speech recognition and synthesis, etc.

[0028] Through the above steps, according to the requirements of the event detection task for the frame extraction frequency, a suitable processor is selected to extract and decode the target video stream, which can avoid excessive occupation of a single processor resource, achieve processor load balancing, and thus improve the processing efficiency of the video stream detection task.

[0029] In a possible implementation, in the above step 102, determining one of a graphics processor GPU and a central processing unit CPU as the first processor according to the frame extraction frequency includes: When the frame extraction frequency is greater than a preset first frequency threshold, the GPU is determined to be the first processor; when the frame extraction frequency is less than or equal to the first frequency threshold, the CPU is determined to be the first processor.

[0030] In an embodiment of the present application, a first frequency threshold can be set. When the frame extraction frequency is greater than the first frequency threshold, the GPU is used as the first processor; when the frame extraction frequency is less than or equal to the first frequency threshold, the CPU is used as the first processor.

[0031] This method can avoid excessive occupation of GPU or CPU resources and achieve processor load balancing.

[0032] In a possible implementation, in the above step 103, the first processor performs frame extraction decoding processing on the target video stream to obtain a video decoding stream, including: The target video stream is subjected to frame extraction and decoding processing by the first processor. During the frame extraction and decoding processing, the resource utilization of the first processor is monitored. When the resource utilization of the first processor reaches a preset threshold, one of the GPU and the CPU is determined to be the second processor according to the first load of the CPU and the second load of the GPU. The target video stream is switched to the second processor for frame extraction and decoding processing to obtain a decoded video stream.

[0033] In an embodiment of the present application, a target video stream is subjected to frame extraction and decoding processing by a first processor (such as a GPU), and during the frame extraction and decoding processing, the resource utilization of the first processor is monitored; when the resource utilization of the first processor reaches a preset threshold (such as 50%), one of the GPU and the CPU is determined to be the second processor based on a first load of the CPU and a second load of the GPU; the target video stream is switched to the second processor to continue the frame extraction and decoding processing to obtain a decoded video stream.

[0034] In this way, when the resource utilization of a processor is high, it can be automatically determined whether the processor needs to be replaced, so as to achieve load balancing between the GPU and the CPU, and further improve the processing efficiency of the video stream detection task.

[0035] The step of determining one of the GPU and the CPU as the second processor according to the first load of the CPU and the second load of the GPU includes: The first load of the CPU is determined according to the total number of frames processed by the CPU and the number of frames occupied by the task; the second load of the GPU is determined according to the total video memory of the GPU, the number of tasks and the video memory of a single task; the ratio between the first load and the second load is determined, and a target product is compared with the number of frames extracted from the target video stream, and one of the GPU and the CPU is determined as the second processor according to the comparison result, wherein the target product is the product of the ratio, the number of frames of the target video stream and a preset load weight.

[0036] In an exemplary embodiment, the first load of the CPU is determined by the following formula: ; Among them, L cpu is the first load, F used is the number of frames occupied by the task, F total is the total number of frames.

[0037] The second load of the GPU is determined by the following formula: ; Among them, L gpu is the second load, N is the number of tasks, S is the single task video memory, and T is the total video memory.

[0038] The second processor is determined by:

[0039] in, is the load weight, FPS video Is the number of frames per second, FPS extraction To increase the frame rate, the load weight can be dynamically updated based on the CPU and GPU load. to adjust task allocation.

[0040] After determining that one of the GPU and the CPU is the second processor according to the first load of the CPU and the second load of the GPU, the method further includes: When the first load is greater than the second load, the frame extraction frequencies of multiple tasks in the CPU are obtained, and the target tasks whose frame extraction frequencies are greater than a preset second frequency threshold among the multiple tasks are transferred to the GPU for frame extraction decoding processing.

[0041] In an exemplary embodiment, if L cpu >L gpu , check the frame extraction frequencies of multiple tasks in the CPU, and transfer the target tasks whose frame extraction frequencies are greater than the second frequency threshold β to the GPU for frame extraction decoding processing, where β can be determined according to actual needs, for example, set to 5.

[0042] In a possible implementation, in the above step 103, the first processor performs frame extraction decoding processing on the target video stream to obtain a video decoding stream, including: The target video stream is subjected to frame extraction decoding processing by the first processor to obtain an initial video decoding stream; the similarity between any two adjacent video frames in the initial video decoding stream is determined; when the similarity is greater than a preset similarity threshold, the target video frame in any two adjacent video frames is filtered out to obtain a video decoding stream.

[0043] In an embodiment of the present application, a GPU or CPU codec frame extraction subprocess is started, and at the same time, information of a target video stream and algorithm task parameters are sent to a video frame extraction subthread to obtain an initial video decoding stream; the startup parameters are called to determine whether to start an image similarity comparison algorithm; if the image similarity comparison algorithm is started, the similarity between any two adjacent video frames in the initial video decoding stream is determined by the similarity comparison algorithm; when the similarity is greater than a preset similarity threshold, the target video frame in any two adjacent video frames is filtered out to obtain a video decoding stream.

[0044] In this way, video frames with relatively small changes compared to the previous frame can be filtered out, which greatly reduces the number of images required to be processed for subsequent event detection tasks and improves the processing efficiency of video stream detection tasks.

[0045] The determining of the similarity between any two adjacent video frames in the initial video decoding stream includes: Acquire multiple video frames in the initial video decoding stream; for each video frame, extract the frequency domain coefficient of the video frame, and determine the hash value corresponding to the video frame according to the average value of the frequency domain coefficient; determine the similarity between any two adjacent video frames according to the Hamming distance between the hash values ​​corresponding to any two adjacent video frames in the multiple video frames.

[0046] In a specific implementation, each video frame can be reduced to 32×32 pixels and then converted to a grayscale image. The grayscale image is subjected to a discrete cosine transform (DCT) to extract the frequency domain coefficients of the video frame, for example, extract the frequency domain coefficients of the 8×8 pixel in the upper left corner. The average value of the frequency domain coefficients is calculated and a hash value (i.e., "fingerprint") is generated. The similarity between adjacent video frames is determined by comparing the Hamming distance of the corresponding hash values ​​of any two adjacent video frames.

[0047] Then, the video decoding stream is sent to the machine learning model through the message queue, and the machine learning model is used to detect the targets and behaviors in the video decoding stream to obtain the event detection results. Among them, the message queue includes Kafka, Redis, and Pular.

[0048] In a possible implementation, in the above step 104, performing the event detection task on the video decoding stream to obtain an event detection result includes: According to the event type of the event detection task, determine the prompt word corresponding to the event type; input the video decoding stream and the prompt word into a pre-trained language model, and detect the video decoding stream according to the prompt word through the pre-trained language model to obtain an event detection result.

[0049] In the embodiment of the present application, the pre-trained language model can be a Qwen2-VL multimodal model deployed using the vllm reasoning framework, and the pre-trained language model is called through the encapsulated interface. Figure 2 As shown, taking the pre-trained language model as a multimodal large model as an example, the above event detection method may include the following steps: Step 201: construct a prompt word library for task detection in different scenarios; Specifically, determine the event type to be detected, such as "fighting", "fire detection", "traffic accident", etc., and design corresponding prompt words for each event category. These prompt words can guide the model to understand and detect related events. For example, for "fighting", the prompt word can be "detect whether the characters in the video have mutual aggression, including but not limited to physical conflicts such as punching and kicking." For the "fire detection" event, the prompt word can be: "Identify whether there are fire features such as flames and smoke in the image or video." For different event detection tasks, customize different prompt statements and store them in the open source database MySQL to obtain a prompt word library for task detection in different scenarios.

[0050] These prompt words not only provide the model with a clear detection target, but also help the model maintain contextual relevance when processing visual and textual data. In this way, multimodal large models can more accurately identify specific events from large amounts of data, thereby improving detection accuracy and efficiency.

[0051] Step 202: assign different tasks, search in the prompt word library, retrieve the prompt word corresponding to the event type, and call the multimodal model matching it; For example, for a fight, the retrieved prompt word is "Detect whether the characters in the video have any attacking behaviors against each other, including but not limited to physical conflicts such as punching and kicking. Return json format data, [{"class":"fighting", "predict":"Ture","bbox": "x1, y1, x2, y2"}", where x1, y1, x2, y2 are the horizontal and vertical coordinates of the upper left corner and the lower right corner of the target respectively.

[0052] Step 203: Perform event detection on the target video stream by calling the multimodal model interface, obtain the data returned by the multimodal model according to the prompt word and the target video stream, determine the event detection result according to these data, and display an alarm for the scene task when the corresponding event is detected; Step 204: The image and the annotated data that generate the alarm are stored. The image is stored in the object storage open source object storage solution mini, and the annotated data is stored in the MySQL database.

[0053] Step 205: Through the data continuously accumulated in the business, the prompt word library can be continuously iterated and improved according to the performance of the model and the feedback of actual application. At the same time, as new event types emerge and the performance of the model is improved, the prompt word library is regularly updated.

[0054] Step 206: Apply the data set generated by the business to the large model for fine-tuning or optimize the detection model for special scenario business.

[0055] In an exemplary embodiment, if Figure 3 As shown, the above-mentioned video stream detection method may include the following steps: Step 301: Access the video according to the scene task; Step 302: Input the video to the codec load balancing module, the codec load balancing module is used to determine the frame extraction frequency according to the event detection task of the target video stream corresponding to the video; according to the frame extraction frequency, determine one of the graphics processor GPU and the central processing unit CPU as the first processor; perform frame extraction decoding processing on the target video stream by the first processor to obtain a video decoding stream; Step 303: putting the video decoding stream into the message queue, and the multimodal model general event detection module performs the event detection task on the video decoding stream in the message queue to obtain the event detection result; Step 304: When it is determined that a relevant event has occurred according to the event detection result, a task alarm is sent.

[0056] Among them, Figure 4 As shown, the above-mentioned codec load balancing module includes a hardware performance evaluation module, a task scheduling module, a codec frame extraction module, a picture similarity filtering module and a message queue; The hardware performance evaluation module includes the CPU performance evaluation component and the GPU performance evaluation component, which are used to perform initial evaluation on the hardware performance of the GPU and CPU, and obtain indicators such as the GPU graphics card model, video memory, and utilization rate, as well as the CPU model, number of cores, and utilization rate; The task scheduling module includes a task allocation algorithm, a task start component, and an exception handling component, which are used to determine the frame extraction frequency according to the event detection task of the target video stream corresponding to the video; according to the frame extraction frequency, determine one of the graphics processor GPU and the central processing unit CPU as the first processor; The codec frame extraction module includes a GPU codec frame extraction component and a CPU codec frame extraction component, which are used to perform frame extraction decoding processing on the target video stream to obtain an initial video decoding stream; The image similarity filtering module is used to determine the similarity between any two adjacent video frames in the initial video decoding stream, and when the similarity is greater than a preset similarity threshold, filter out the target video frame in any two adjacent video frames to obtain the video decoding stream; Message queue, used to store video decoding stream.

[0057] The embodiment of the present application provides a video stream detection method, which determines the frame extraction frequency according to the event detection task of the target video stream; determines one of the graphics processor GPU and the central processing unit CPU as the first processor according to the frame extraction frequency; performs frame extraction and decoding processing on the target video stream by the first processor to obtain a video decoding stream; and performs the event detection task on the video decoding stream to obtain the event detection result. In this way, according to the demand of the event detection task for the frame extraction frequency, a suitable processor is selected to perform frame extraction and decoding on the target video stream, which can avoid excessive occupation of a single processor resource, realize processor load balancing, and thus improve the processing efficiency of the video stream detection task.

[0058] In addition, after the video stream is extracted and decoded, an image similarity comparison algorithm is added to filter similar images, thereby greatly reducing the computing power requirements of the multimodal model; by constructing a prompt vocabulary library for different event detections and combining the capabilities of the multimodal model, the detection of various scene events can be effectively realized; through the accumulation of business data for scene event detection, the prompt vocabulary library is iteratively optimized using a semi-supervised method, and the large model is fine-tuned or the small model is optimized for special scene businesses, so as to further improve the accuracy of event detection corresponding to the video stream detection task.

[0059] Figure 5 The schematic diagram of the structure of the video stream detection device provided in the embodiment of the present application is shown. The video stream detection device can be implemented as follows: Figure 1 or Figure 3 In the embodiment shown, all or part of the contents, the video stream detection device 500 includes: The frame extraction frequency determination module 510 is used to determine the frame extraction frequency according to the event detection task of the target video stream; A processor determination module 520, configured to determine, according to the frame extraction frequency, one of a graphics processor GPU and a central processing unit CPU as a first processor; A frame extraction decoding processing module 530, configured to perform frame extraction decoding processing on the target video stream through the first processor to obtain a video decoding stream; The detection result determination module 540 is used to perform the event detection task on the video decoding stream to obtain an event detection result.

[0060] In a possible implementation, the processor determination module 520, when used to determine, according to the frame extraction frequency, that one of a graphics processor GPU and a central processing unit CPU is the first processor, is specifically used to: When the frame extraction frequency is greater than a preset first frequency threshold, determining that the GPU is a first processor; When the frame extraction frequency is less than or equal to the first frequency threshold, the CPU is determined to be the first processor.

[0061] In a possible implementation, the frame extraction decoding processing module 530, when used to perform frame extraction decoding processing on the target video stream through the first processor to obtain a video decoding stream, is specifically used to: Performing frame extraction and decoding processing on the target video stream by the first processor, and monitoring resource utilization of the first processor during the frame extraction and decoding processing; When the resource utilization rate of the first processor reaches a preset threshold, determining one of the GPU and the CPU as the second processor according to a first load of the CPU and a second load of the GPU; The target video stream is switched to the second processor for frame extraction and decoding processing to obtain a video decoding stream.

[0062] The frame extraction decoding processing module 530 is further used for: When the first load is greater than the second load, the frame extraction frequencies of multiple tasks in the CPU are obtained, and the target tasks whose frame extraction frequencies are greater than a preset second frequency threshold among the multiple tasks are transferred to the GPU for frame extraction decoding processing.

[0063] In a possible implementation, the frame extraction decoding processing module 530, when used to perform frame extraction decoding processing on the target video stream through the first processor to obtain a video decoding stream, is specifically used to: Performing frame extraction decoding processing on the target video stream by the first processor to obtain an initial video decoding stream; Determining the similarity between any two adjacent video frames in the initial video decoding stream; When the similarity is greater than a preset similarity threshold, the target video frame in any two adjacent video frames is filtered out to obtain a video decoding stream.

[0064] In a possible implementation, the detection result determination module 540, when used to perform the event detection task on the video decoding stream to obtain the event detection result, is specifically used to: Determining, according to the event type of the event detection task, a prompt word corresponding to the event type; The video decoding stream and the prompt word are input into a pre-trained language model, and the video decoding stream is detected according to the prompt word by the pre-trained language model to obtain an event detection result.

[0065] The embodiment of the present application provides a video stream detection device, including a frame extraction frequency determination module, a processor determination module, a frame extraction decoding processing module and a detection result determination module; the frame extraction frequency determination module determines the frame extraction frequency according to the event detection task of the target video stream; the processor determination module determines one of the graphics processor GPU and the central processing unit CPU as the first processor according to the frame extraction frequency; the frame extraction decoding processing module performs frame extraction decoding processing on the target video stream through the first processor to obtain a video decoding stream; the detection result determination module performs the event detection task on the video decoding stream to obtain an event detection result. In this way, according to the demand of the event detection task for the frame extraction frequency, a suitable processor is selected to perform frame extraction decoding on the target video stream, which can avoid excessive occupation of a single processor resource, realize processor load balancing, and thus improve the processing efficiency of the video stream detection task.

[0066] Figure 6 The hardware structure diagram of the electronic device provided by the embodiment of the present application is shown. Referring to the figure, at the hardware level, the electronic device 600 includes a processor 610, and optionally, an internal bus 620, a network interface 630, and a memory 640. Among them, the memory 640 may include a memory 641, such as a high-speed random access memory (Random-Access Memory, RAM), and may also include a non-volatile memory 642 (non-volatile memory), such as at least one disk storage, etc. Of course, the electronic device may also include hardware required for other services.

[0067] The processor 610, the network interface 630 and the memory may be interconnected via an internal bus 620, which may be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The bus may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one bidirectional arrow is used in the figure, but it does not mean that there is only one bus or one type of bus.

[0068] The memory 640 stores programs. Specifically, the programs may include program codes, and the program codes include computer operation instructions. The memory 640 may include a memory 641 and a non-volatile memory 642, and provide instructions and data to the processor 610.

[0069] The processor 610 reads the corresponding computer program from the non-volatile memory 642 into the memory and then runs it, forming a device for locating the target user at the logical level. The processor 610 executes the program stored in the memory and specifically performs: Figure 1 or Figure 4 The method disclosed in the illustrated embodiment realizes the functions and beneficial effects of each method described in the foregoing method embodiments, which will not be described in detail here.

[0070] The above application Figure 1 or Figure 4 The method disclosed in the illustrated embodiment can be applied to the processor 610 or implemented by the processor 610. The processor 610 may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by the hardware integrated logic circuit or software instructions in the processor 610. The above processor 610 can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The methods, steps and logic block diagrams disclosed in the embodiments of the present application can be implemented or executed. The general processor can be a microprocessor or the processor can also be any conventional processor, etc. The steps of the method disclosed in the embodiments of the present application can be directly embodied as a hardware decoding processor for execution, or a combination of hardware and software modules in the decoding processor for execution. The software module can be located in a storage medium mature in the art, such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory, or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory, and the processor reads the information in the memory and completes the steps of the above method in combination with its hardware.

[0071] The computer device can also execute the methods described in the foregoing method embodiments, and realize the functions and beneficial effects of the methods described in the foregoing method embodiments, which will not be repeated here.

[0072] Of course, in addition to software implementation methods, the electronic device of the present application does not exclude other implementation methods, such as logic devices or a combination of software and hardware, etc. That is to say, the execution subject of the following processing flow is not limited to each logic unit, but can also be hardware or logic devices.

[0073] The embodiment of the present application also provides a computer-readable storage medium, wherein the computer-readable medium stores one or more programs, and when the one or more programs are executed by an electronic device including multiple application programs, the electronic device executes Figure 1 or Figure 4 The method disclosed in the illustrated embodiment realizes the functions and beneficial effects of each method described in the foregoing method embodiments, which will not be described in detail here.

[0074] The computer-readable storage medium includes a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.

[0075] Furthermore, an embodiment of the present application also provides a computer program product, the computer program product comprising a computer program stored on a non-transitory computer-readable storage medium, the computer program comprising program instructions, and when the program instructions are executed by a computer, the following process is implemented: Figure 1 or Figure 4 The method disclosed in the illustrated embodiment realizes the functions and beneficial effects of each method described in the foregoing method embodiments, which will not be described in detail here.

[0076] The embodiments of the present application can be applied to various electronic device collaboration or interconnection scenarios, including: collaboration and interconnection between mobile phones and laptops / tablet computers; collaboration and interconnection between mobile terminals and smart TVs / displays; collaboration and interconnection between mobile phones or tablet computers and in-car entertainment systems; collaboration and interconnection between mobile terminals and smart conference systems, etc., so as to meet the needs of users in various scenarios such as smart homes, smart offices, and smart travel.

[0077] In short, the above is only a preferred embodiment of the present application and does not limit the protection scope of the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

[0078] The systems, devices, modules or units described in the above embodiments may be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer. Specifically, the computer may be, for example, a personal computer, a laptop computer, a cellular phone, a camera phone, a smart phone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, a wearable device, or a combination of any of these devices.

[0079] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape disk storage or other magnetic storage devices or any other non-transmission media that can store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0080] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0081] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

Claims

1. A video stream detection method, characterized in that: include: Determine the frame extraction frequency according to the event detection task of the target video stream; According to the frame extraction frequency, determining one of a graphics processor GPU and a central processing unit CPU as a first processor; Performing frame extraction and decoding processing on the target video stream by the first processor to obtain a video decoding stream; The event detection task is performed on the video decoding stream to obtain an event detection result.

2. The method according to claim 1, characterized in that The step of determining one of a graphics processor GPU and a central processing unit CPU as a first processor according to the frame extraction frequency includes: When the frame extraction frequency is greater than a preset first frequency threshold, determining that the GPU is a first processor; When the frame extraction frequency is less than or equal to the first frequency threshold, the CPU is determined to be the first processor.

3. The method according to claim 1, characterized in that: The step of performing frame extraction decoding processing on the target video stream by the first processor to obtain a video decoding stream includes: Performing frame extraction and decoding processing on the target video stream by the first processor, and monitoring resource utilization of the first processor during the frame extraction and decoding processing; When the resource utilization rate of the first processor reaches a preset threshold, determining one of the GPU and the CPU as the second processor according to a first load of the CPU and a second load of the GPU; The target video stream is switched to the second processor for frame extraction and decoding processing to obtain a video decoding stream.

4. The method according to claim 3, characterized in that After determining, according to the first load of the CPU and the second load of the GPU, that one of the GPU and the CPU is the second processor, the method further includes: When the first load is greater than the second load, the frame extraction frequencies of multiple tasks in the CPU are obtained, and the target tasks whose frame extraction frequencies are greater than a preset second frequency threshold among the multiple tasks are transferred to the GPU for frame extraction decoding processing.

5. The method according to claim 1, characterized in that The step of performing frame extraction decoding processing on the target video stream by the first processor to obtain a video decoding stream includes: Performing frame extraction decoding processing on the target video stream by the first processor to obtain an initial video decoding stream; Determining the similarity between any two adjacent video frames in the initial video decoding stream; When the similarity is greater than a preset similarity threshold, the target video frame in any two adjacent video frames is filtered out to obtain a video decoding stream.

6. The method according to any one of claims 1 to 5, characterized in that: The performing the event detection task on the video decoding stream to obtain an event detection result includes: Determining, according to the event type of the event detection task, a prompt word corresponding to the event type; The video decoding stream and the prompt word are input into a pre-trained language model, and the video decoding stream is detected according to the prompt word by the pre-trained language model to obtain an event detection result.

7. A video stream detection device, characterized in that: include: A frame extraction frequency determination module is used to determine the frame extraction frequency according to the event detection task of the target video stream; A processor determination module, used for determining, according to the frame extraction frequency, one of a graphics processor GPU and a central processing unit CPU as a first processor; A frame extraction decoding processing module, used for performing frame extraction decoding processing on the target video stream through the first processor to obtain a video decoding stream; The detection result determination module is used to perform the event detection task on the video decoding stream to obtain an event detection result.

8. An electronic device, characterized in that: The electronic device comprises a processor and a memory, wherein the memory stores a program or instruction that can be run on the processor, and when the program or instruction is executed by the processor, the steps of the method according to any one of claims 1 to 6 are implemented.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a program or instruction, and when the program or instruction is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

10. A computer program product, characterized in that The computer program product comprises a computer program stored on a non-transitory computer-readable storage medium, wherein the computer program comprises program instructions, and when the program instructions are executed by a computer, the computer is caused to perform the steps of the method according to any one of claims 1 to 6.

Images