Video stream detection method, device, equipment, storage medium and program product

Abstract

This application discloses a video stream detection method, apparatus, device, storage medium, and program product, belonging to the field of video stream processing technology. The method comprises: determining a frame extraction frequency based on an event detection task for a target video stream; determining, based on the frame extraction frequency, one of a graphics processing unit (GPU) and a central processing unit (CPU) as a first processor; performing frame extraction and decoding processing on the target video stream by the first processor to obtain a decoded video stream; and executing an event detection task on the decoded video stream to obtain an event detection result. This approach avoids excessive resource usage of a single processor, achieves processor load balancing, and thereby improves the processing efficiency of the video stream detection task.

Description

Technical Field

[0001] The present application relates to the field of video stream processing technology, and in particular to a video stream detection method, apparatus, device, storage medium and program product. Background Art

[0002] Video stream encoding, decoding, and frame extraction are core technologies in video processing. They primarily involve encoding and decoding video streams, as well as extracting specific frames for subsequent processing or analysis. This technology is widely used in fields such as video surveillance, content review, media playback, and video conferencing. Currently, video stream encoding, decoding, and frame extraction solutions fall into two main categories: those based on the Central Processing Unit (CPU) and those based on the Graphics Processing Unit (GPU).

[0003] Existing solutions usually rely on a single hardware resource (such as the CPU or GPU) to perform video frame extraction and event detection tasks, without fully considering the system resource usage and remaining computing power, resulting in excessive hardware resource usage, which in turn affects the processing efficiency of video stream detection tasks. Summary of the Invention

[0004] The embodiments of the present application provide a video stream detection method, apparatus, device, storage medium, and program product to at least solve the problem of low processing efficiency of video stream detection tasks in related technologies.

[0005] In order to solve the above technical problems, this application is implemented as follows:

[0006] In a first aspect, an embodiment of the present application provides a video stream detection method, comprising: determining a frame extraction frequency according to an event detection task of a target video stream; determining one of a graphics processing unit (GPU) and a central processing unit (CPU) as a first processor according to the frame extraction frequency; performing frame extraction decoding processing on the target video stream by the first processor to obtain a video decoding stream; executing the event detection task on the video decoding stream to obtain an event detection result.

[0007] In the second aspect, an embodiment of the present application provides a video stream detection device, including: a frame extraction frequency determination module, used to determine the frame extraction frequency according to the event detection task of the target video stream; a processor determination module, used to determine one of the graphics processing unit GPU and the central processing unit CPU as the first processor according to the frame extraction frequency; a frame extraction decoding processing module, used to perform frame extraction decoding processing on the target video stream through the first processor to obtain a video decoding stream; a detection result determination module, used to execute the event detection task on the video decoding stream to obtain an event detection result.

[0008] In a third aspect, an embodiment of the present application provides an electronic device, comprising a processor and a memory, wherein the memory stores programs or instructions that can be run on the processor, and when the programs or instructions are executed by the processor, the steps of the method described in the first aspect above are implemented.

[0009] In a fourth aspect, an embodiment of the present application provides a readable storage medium, on which a program or instruction is stored. When the program or instruction is executed by a processor, the steps of the method described in the first aspect above are implemented.

[0010] In a fifth aspect, an embodiment of the present application provides a computer program product, which includes a computer program stored on a non-transitory computer-readable storage medium, and the computer program includes program instructions. When the program instructions are executed by a computer, the computer performs the steps of the method described in the first aspect above.

[0011] In an embodiment of the present application, a frame extraction frequency is determined based on an event detection task for a target video stream; based on the frame extraction frequency, one of a graphics processing unit (GPU) and a central processing unit (CPU) is determined as a first processor; the first processor performs frame extraction and decoding processing on the target video stream to obtain a decoded video stream; and the event detection task is performed on the decoded video stream to obtain an event detection result. In this way, based on the frame extraction frequency requirements of the event detection task, an appropriate processor is selected to perform frame extraction and decoding on the target video stream, thereby avoiding excessive occupation of a single processor's resources, achieving processor load balancing, and thus improving the processing efficiency of the video stream detection task.

[0012] It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0014] Figure 1 A schematic diagram of a flow chart of a video stream detection method provided in an embodiment of the present application is shown;

[0015] Figure 2 A schematic diagram of a process flow of an event detection method provided in an embodiment of the present application is shown;

[0016] Figure 3 Another schematic diagram of a video stream detection method according to an embodiment of the present invention is shown;

[0017] Figure 4 Shown Figure 3Schematic diagram of the processing flow of the codec load balancing module;

[0018] Figure 5 A schematic diagram of the structure of a video stream detection device provided in an embodiment of the present application is shown;

[0019] Figure 6 A schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present application is shown. DETAILED DESCRIPTION

[0020] Exemplary embodiments will be described in detail herein, with examples illustrated in the accompanying drawings. In the following description, when referring to the drawings, identical numerals in different figures represent identical or similar elements, unless otherwise indicated. The embodiments described in the following exemplary embodiments are not intended to represent all embodiments consistent with the present application. Rather, they are merely examples of apparatus and methods consistent with certain aspects of the present application, as detailed in the appended claims.

[0021] Currently, video stream encoding, decoding, and frame extraction solutions are primarily categorized as CPU-based and GPU-based. CPU-based frame extraction solutions typically utilize the Fast Forward Moving Picture Experts Group (FFmpeg) tool to extract frames from the video stream and save them as images. Artificial intelligence (AI) algorithms then read the images, perform pre-processing, inference, and post-processing, and ultimately upload the results. Compared to CPU-based solutions, GPU-based frame extraction solutions can significantly improve encoding and decoding speeds when applied to AI algorithms, enabling more efficient task completion.

[0022] Related solutions propose that video stream data encoding and decoding, video processing, and AI inference tasks are all completed within the GPU, preventing the video stream from flowing outside the GPU and making the entire video stream inference system more functional. However, a single hardware resource (such as a CPU or GPU) may encounter computing bottlenecks when processing the entire video stream encoding and decoding and AI inference process, resulting in excessive resource utilization and affecting the processing efficiency of video stream detection tasks.

[0023] In response to the problems existing in the above-mentioned video stream detection process, an embodiment of the present application provides a video stream detection method. This method selects a suitable processor to perform frame extraction and decoding on the target video stream based on the frame extraction frequency requirements of the event detection task, so as to avoid excessive occupation of single processor resources, achieve processor load balancing, and thus improve the processing efficiency of the video stream detection task.

[0024] Figure 1The flowchart of the video stream detection method provided in the embodiment of the present application is shown. The execution subject of the method can be a terminal device or a server, wherein the terminal device can be a device such as a personal computer, or a mobile terminal device such as a mobile phone or a tablet computer, and the terminal device can be a terminal device used by a user. The server can be an independent server, or a server cluster composed of multiple servers. Moreover, the server can be a background server of a certain business, or a background server of a certain platform or application (such as a security monitoring system, an intelligent transportation system, a video analysis platform, a smart city platform, etc.). In the embodiment of the present application, the execution subject is taken as an example to illustrate. For the case of the terminal device, it can be processed according to the following relevant content, which will not be repeated here. As shown in the figure, the method 100 can include the following steps:

[0025] Step 101: Determine the frame extraction frequency according to the event detection task of the target video stream.

[0026] In a specific implementation, the target video stream of a preset area can be collected in real time by monitoring equipment such as cameras, or a pre-stored target video stream to be detected can be obtained, and an event detection task for the target video stream can be set. For example, fighting incidents, fire detection events, traffic accidents, traffic flow, etc. can be detected in the target video stream. Among them, the same target video stream can correspond to multiple event detection tasks.

[0027] The frame extraction frequency is determined based on the event detection task. For example, to detect fights in the target video, assuming the frame rate of the target video stream is 30 frames per second (fps), the frame extraction frequency can be set to 10 frames per second or higher. This ensures that the key actions of the fight are captured in a timely manner.

[0028] Step 102: Determine one of a graphics processing unit (GPU) and a central processing unit (CPU) as a first processor according to the frame extraction frequency.

[0029] In specific implementations, the GPU is much more efficient than the CPU in performing frame extraction and decoding of a single video stream, but the video memory resources occupied by the GPU frame extraction and decoding do not depend on the frame extraction frequency, but are only related to the video stream's own properties such as resolution; the CPU frame extraction and decoding only occupies resources during the current call, and the resources will be released after processing. The CPU uses ffmpeg to extract and decode the video stream, and makes interval calls at the frequency required by the task. After each call, it extracts a picture and releases the resources. Therefore, the GPU is more suitable for tasks that require a higher frequency of video stream frame extraction, and the CPU is suitable for tasks that require a lower frequency of video stream frame extraction. According to the frame extraction frequency corresponding to the event detection task, one of the GPU and CPU can be determined as the first processor. For example, when the frame extraction frequency f>a, the GPU is used for processing; when the frame extraction frequency f≤a, the GPU is used for processing, and a can be set according to the processor's resources.

[0030] Step 103: The first processor performs frame extraction decoding on the target video stream to obtain a decoded video stream.

[0031] In a specific implementation, a selected first processor (such as a GPU) extracts frames from the target video stream and decodes each extracted frame. The decoded frames are converted into standard image data, resulting in a decoded video stream for subsequent event detection tasks. For example, in a fight scene, the GPU would decode the extracted frames into image data, providing input for subsequent object detection algorithms (such as action recognition and event detection).

[0032] Step 104: Execute the event detection task on the video decoding stream to obtain an event detection result.

[0033] In practice, the decoded video stream can be fed into a machine learning model, which can then detect objects and behaviors in the decoded video stream to generate event detection results. Examples of machine learning models include large multimodal models, object detection (e.g., YOLO, Faster R-CNN), action recognition (based on convolutional neural networks (CNNs), or time series analysis models (e.g., long short-term memory (LSTM) networks).

[0034] A multimodal large model is a machine learning model that can understand and process multiple types of data (also known as "modalities"), including text, image, audio, and video. This model can integrate information from multiple modalities and extract useful insights to perform more complex and intelligent tasks, such as visual question answering (AI interviewers), image-to-text generation, and speech recognition and synthesis.

[0035] Through the above steps, according to the frame extraction frequency requirements of the event detection task, a suitable processor is selected to extract and decode the target video stream. This can avoid excessive occupation of single processor resources, achieve processor load balancing, and thus improve the processing efficiency of the video stream detection task.

[0036] In a possible implementation, in step 102, determining one of a graphics processor (GPU) and a central processing unit (CPU) as the first processor according to the frame extraction frequency includes:

[0037] When the frame extraction frequency is greater than a preset first frequency threshold, the GPU is determined to be the first processor; when the frame extraction frequency is less than or equal to the first frequency threshold, the CPU is determined to be the first processor.

[0038] In an embodiment of the present application, a first frequency threshold can be set. When the frame extraction frequency is greater than the first frequency threshold, the GPU is used as the first processor; when the frame extraction frequency is less than or equal to the first frequency threshold, the CPU is used as the first processor.

[0039] This method can avoid excessive occupation of GPU or CPU resources and achieve processor load balancing.

[0040] In a possible implementation, in step 103, performing frame extraction decoding on the target video stream by the first processor to obtain a decoded video stream includes:

[0041] The target video stream is subjected to frame extraction and decoding processing by the first processor. During the frame extraction and decoding processing, the resource utilization of the first processor is monitored. When the resource utilization of the first processor reaches a preset threshold, one of the GPU and the CPU is determined to be the second processor based on the first load of the CPU and the second load of the GPU. The target video stream is switched to the second processor for frame extraction and decoding processing to obtain a video decoding stream.

[0042] In an embodiment of the present application, a target video stream is subjected to frame extraction and decoding processing by a first processor (such as a GPU). During the frame extraction and decoding processing, the resource utilization of the first processor is monitored; when the resource utilization of the first processor reaches a preset threshold (such as 50%), one of the GPU and the CPU is determined to be the second processor based on a first load of the CPU and a second load of the GPU; the target video stream is switched to the second processor to continue the frame extraction and decoding processing to obtain a video decoded stream.

[0043] In this way, when the resource utilization of a certain processor is high, it can be automatically determined whether the processor needs to be replaced, so as to achieve load balancing between the GPU and CPU, and further improve the processing efficiency of the video stream detection task.

[0044] The step of determining one of the GPU and the CPU as the second processor based on the first load of the CPU and the second load of the GPU includes:

[0045] The first load of the CPU is determined based on the total number of frames processed by the CPU and the number of frames occupied by the task; the second load of the GPU is determined based on the total video memory, the number of tasks and the video memory of a single task of the GPU; the ratio between the first load and the second load is determined, and a target product is compared with the number of frames extracted from the target video stream. According to the comparison result, one of the GPU and the CPU is determined to be the second processor, wherein the target product is the product of the ratio, the number of picture frames of the target video stream and a preset load weight.

[0046] In an exemplary embodiment, the first load of the CPU is determined by the following formula:

[0047] ;

[0048] Among them, L cpu is the first load, F used is the number of frames occupied by the task, F total is the total number of frames.

[0049] The second load of the GPU is determined by the following formula:

[0050] ;

[0051] Among them, L gpu is the second load, N is the number of tasks, S is the single task video memory, and T is the total video memory.

[0052] The second processor is determined by:

[0053]

[0054] in, is the load weight, FPS video For the number of frames per second, FPS extraction To increase the frame rate, the load weight can be dynamically updated based on the CPU and GPU load. to adjust task allocation.

[0055] After determining that one of the GPU and the CPU is the second processor according to the first load of the CPU and the second load of the GPU, the method further includes:

[0056] When the first load is greater than the second load, the frame extraction frequencies of multiple tasks in the CPU are obtained, and the target tasks whose frame extraction frequencies are greater than a preset second frequency threshold among the multiple tasks are transferred to the GPU for frame extraction decoding processing.

[0057] In an exemplary embodiment, if L cpu >L gpu , check the frame extraction frequencies of multiple tasks in the CPU, and transfer the target tasks whose frame extraction frequencies are greater than the second frequency threshold β to the GPU for frame extraction decoding processing, where β can be determined according to actual needs, for example, set to 5.

[0058] In a possible implementation, in step 103, performing frame extraction decoding on the target video stream by the first processor to obtain a decoded video stream includes:

[0059] The target video stream is subjected to frame extraction decoding processing by the first processor to obtain an initial video decoding stream; the similarity between any two adjacent video frames in the initial video decoding stream is determined; and when the similarity is greater than a preset similarity threshold, the target video frame in any two adjacent video frames is filtered out to obtain a video decoding stream.

[0060] In an embodiment of the present application, a GPU or CPU codec frame extraction subprocess is started, and at the same time, information of the target video stream and algorithm task parameters are sent to the video frame extraction subthread to obtain an initial video decoding stream; the startup parameters are called to determine whether to start the image similarity comparison algorithm; if the image similarity comparison algorithm is started, the similarity between any two adjacent video frames in the initial video decoding stream is determined by the similarity comparison algorithm; when the similarity is greater than a preset similarity threshold, the target video frame in any two adjacent video frames is filtered out to obtain a video decoding stream.

[0061] In this way, video frames with relatively small changes compared to the previous frame can be filtered out, which greatly reduces the number of images required to be processed for subsequent event detection tasks and improves the processing efficiency of video stream detection tasks.

[0062] The determining of the similarity between any two adjacent video frames in the initial video decoding stream includes:

[0063] Acquire multiple video frames from the initial video decoding stream; for each video frame, extract the frequency domain coefficients of the video frame, and determine a hash value corresponding to the video frame based on an average value of the frequency domain coefficients; and determine the similarity between any two adjacent video frames based on the Hamming distance between the hash values ​​corresponding to any two adjacent video frames in the multiple video frames.

[0064] In practice, each video frame can be scaled down to 32×32 pixels and then converted to a grayscale image. A discrete cosine transform (DCT) is then performed on the grayscale image to extract the frequency domain coefficients of the video frame. For example, the frequency domain coefficients of the upper left corner 8×8 pixels are extracted. The average of these frequency domain coefficients is calculated, and a hash value (also known as a "fingerprint") is generated. The similarity between adjacent video frames is determined by comparing the Hamming distance between the corresponding hash values ​​of any two adjacent video frames.

[0065] The video decoded stream is then sent to the machine learning model via a message queue. The machine learning model detects objects and behaviors in the video decoded stream and obtains event detection results. Message queues include Kafka, Redis, and Pular.

[0066] In a possible implementation, in step 104, performing the event detection task on the video decoding stream to obtain an event detection result includes:

[0067] According to the event type of the event detection task, a prompt word corresponding to the event type is determined; the video decoding stream and the prompt word are input into a pre-trained language model, and the video decoding stream is detected according to the prompt word by the pre-trained language model to obtain an event detection result.

[0068] In the embodiment of the present application, the pre-trained language model can be a Qwen2-VL multimodal model deployed using the vllm reasoning framework, and the pre-trained language model is called through the encapsulated interface. Figure 2 As shown, taking the pre-trained language model as a multimodal large model as an example, the above event detection method may include the following steps:

[0069] Step 201: Construct a prompt word library for task detection in different scenarios;

[0070] Specifically, we determine the event type to detect, such as "fight," "fire detection," or "traffic accident." We then design corresponding prompts for each event category. These prompts guide the model in understanding and detecting related events. For example, for "fight," a prompt might be "Detect whether characters in the video are engaging in aggressive behavior, including but not limited to physical altercations like punching and kicking." For "fire detection," a prompt might be "Identify whether fire characteristics, such as flames or smoke, appear in the image or video." We customize prompt statements for different event detection tasks and store them in the open-source MySQL database, creating a library of prompts tailored to different task scenarios.

[0071] These cues not only provide the model with a clear detection target but also help it maintain contextual relevance when processing visual and textual data. In this way, large multimodal models can more accurately identify specific events from large amounts of data, thereby improving detection accuracy and efficiency.

[0072] Step 202: Assign different tasks, search the prompt word library, retrieve the prompt word corresponding to the event type, and call the multimodal model that matches it;

[0073] For example, for a fighting incident, the retrieved prompt word is "Detect whether the characters in the video have any attacking behaviors against each other, including but not limited to physical conflicts such as punching and kicking. Return json format data, [{"class":"fighting", "predict":"Ture","bbox": "x1, y1, x2, y2"}", where x1, y1, x2, y2 are the horizontal and vertical coordinates of the upper left corner and the lower right corner of the target respectively.

[0074] Step 203: Perform event detection on the target video stream by calling the multimodal model interface, obtain the data returned by the multimodal model based on the prompt word and the target video stream, determine the event detection result based on this data, and display an alarm for the scene task when the corresponding event is detected;

[0075] Step 204: The image and the annotation data of the alarm are stored. The image is stored in the object storage open source object storage solution mini, and the annotation data is stored in the MySQL database.

[0076] Step 205: Through the data continuously accumulated in the business, the prompt word library can be continuously iterated and improved according to the performance of the model and the feedback of actual application. At the same time, the prompt word library is regularly updated as new event types emerge and the performance of the model improves.

[0077] Step 206: Apply the data set generated by the business to the large model for fine-tuning or optimize the detection model for special scenario business.

[0078] In an exemplary embodiment, Figure 3 As shown, the above-mentioned video stream detection method may include the following steps:

[0079] Step 301: Access the video according to the scenario task;

[0080] Step 302: Input the video to a codec load balancing module. The codec load balancing module is configured to determine a frame extraction frequency based on an event detection task of a target video stream corresponding to the video; determine, based on the frame extraction frequency, one of a graphics processing unit (GPU) and a central processing unit (CPU) as a first processor; and perform frame extraction and decoding processing on the target video stream by the first processor to obtain a decoded video stream.

[0081] Step 303: The video decoding stream is placed in a message queue, and the multimodal model general event detection module performs an event detection task on the video decoding stream in the message queue to obtain an event detection result;

[0082] Step 304: When it is determined that a related event has occurred according to the event detection result, a task alarm is sent.

[0083] Among them, such as Figure 4 As shown, the above-mentioned codec load balancing module includes a hardware performance evaluation module, a task scheduling module, a codec frame extraction module, a picture similarity filtering module and a message queue;

[0084] The hardware performance evaluation module includes CPU performance evaluation components and GPU performance evaluation components, which are used to perform initial evaluation of GPU and CPU hardware performance, and obtain indicators such as GPU graphics card model, video memory, utilization rate, and CPU model, number of cores, and utilization rate;

[0085] The task scheduling module includes a task allocation algorithm, a task startup component, and an exception handling component, and is used to determine the frame extraction frequency according to the event detection task of the target video stream corresponding to the video; according to the frame extraction frequency, determine one of the graphics processor GPU and the central processing unit CPU as the first processor;

[0086] The codec frame extraction module includes a GPU codec frame extraction component and a CPU codec frame extraction component, which is used to perform frame extraction and decoding processing on the target video stream to obtain an initial video decoding stream;

[0087] An image similarity filtering module is used to determine the similarity between any two adjacent video frames in the initial video decoding stream, and when the similarity is greater than a preset similarity threshold, filter out the target video frame from any two adjacent video frames to obtain a video decoding stream;

[0088] Message queue, used to store video decoding stream.

[0089] The present application provides a video stream detection method that determines a frame extraction frequency based on an event detection task for a target video stream; determines, based on the frame extraction frequency, one of a graphics processing unit (GPU) and a central processing unit (CPU) as a first processor; uses the first processor to perform frame extraction and decoding processing on the target video stream to obtain a decoded video stream; and executes an event detection task on the decoded video stream to obtain an event detection result. In this way, based on the frame extraction frequency requirements of the event detection task, an appropriate processor is selected to perform frame extraction and decoding on the target video stream, thereby avoiding excessive occupation of a single processor's resources, achieving processor load balancing, and thus improving the processing efficiency of the video stream detection task.

[0090] In addition, after the video stream is extracted and decoded, an image similarity comparison algorithm is added to filter similar images, thereby greatly reducing the computing power requirements of the multimodal model; by constructing a prompt word library for different event detection and combining the capabilities of the multimodal model, it is possible to effectively detect various scene events; through the accumulation of business data for scene event detection, the prompt word library is iteratively optimized using a semi-supervised method, and the large model is fine-tuned or the small model is optimized for special scene businesses, so as to further improve the accuracy of event detection corresponding to the video stream detection task.

[0091] Figure 5 The schematic diagram of the structure of the video stream detection device provided by the embodiment of the present application is shown. The video stream detection device can be implemented as follows Figure 1 or Figure 3 In the embodiment shown, all or part of the content, the video stream detection device 500 includes:

[0092] The frame extraction frequency determination module 510 is used to determine the frame extraction frequency according to the event detection task of the target video stream;

[0093] A processor determination module 520 is configured to determine, based on the frame extraction frequency, one of a graphics processor (GPU) and a central processing unit (CPU) as a first processor;

[0094] a frame extraction decoding processing module 530, configured to perform frame extraction decoding processing on the target video stream through the first processor to obtain a video decoded stream;

[0095] The detection result determination module 540 is configured to perform the event detection task on the video decoding stream to obtain an event detection result.

[0096] In a possible implementation, the processor determination module 520, when used to determine, based on the frame extraction frequency, that one of a graphics processor GPU and a central processing unit CPU is the first processor, is specifically configured to:

[0097] When the frame extraction frequency is greater than a preset first frequency threshold, determining that the GPU is the first processor;

[0098] When the frame extraction frequency is less than or equal to the first frequency threshold, the CPU is determined to be the first processor.

[0099] In a possible implementation, the frame extraction decoding processing module 530, when configured to perform frame extraction decoding processing on the target video stream through the first processor to obtain a video decoded stream, is specifically configured to:

[0100] performing frame extraction and decoding processing on the target video stream by the first processor, and monitoring resource utilization of the first processor during the frame extraction and decoding processing;

[0101] When the resource utilization rate of the first processor reaches a preset threshold, determining one of the GPU and the CPU as the second processor according to a first load of the CPU and a second load of the GPU;

[0102] The target video stream is switched to the second processor for frame extraction and decoding processing to obtain a video decoding stream.

[0103] The frame extraction decoding processing module 530 is further configured to:

[0104] When the first load is greater than the second load, the frame extraction frequencies of multiple tasks in the CPU are obtained, and the target tasks whose frame extraction frequencies are greater than a preset second frequency threshold among the multiple tasks are transferred to the GPU for frame extraction decoding processing.

[0105] In a possible implementation, the frame extraction decoding processing module 530, when configured to perform frame extraction decoding processing on the target video stream through the first processor to obtain a video decoded stream, is specifically configured to:

[0106] Performing frame extraction decoding on the target video stream by the first processor to obtain an initial video decoding stream;

[0107] Determining the similarity between any two adjacent video frames in the initial video decoded stream;

[0108] When the similarity is greater than a preset similarity threshold, the target video frame in any two adjacent video frames is filtered out to obtain a video decoding stream.

[0109] In a possible implementation, the detection result determination module 540, when performing the event detection task on the video decoding stream and obtaining the event detection result, is specifically configured to:

[0110] Determining, according to the event type of the event detection task, a prompt word corresponding to the event type;

[0111] The video decoding stream and the prompt word are input into a pre-trained language model, and the video decoding stream is detected according to the prompt word by the pre-trained language model to obtain an event detection result.

[0112] The embodiment of the present application provides a video stream detection device, including a frame extraction frequency determination module, a processor determination module, a frame extraction decoding processing module and a detection result determination module; the frame extraction frequency determination module determines the frame extraction frequency according to the event detection task of the target video stream; the processor determination module determines one of the graphics processing unit (GPU) and the central processing unit (CPU) as the first processor according to the frame extraction frequency; the frame extraction decoding processing module performs frame extraction decoding processing on the target video stream through the first processor to obtain a video decoding stream; the detection result determination module performs the event detection task on the video decoding stream to obtain an event detection result. In this way, according to the requirements of the event detection task for the frame extraction frequency, a suitable processor is selected to perform frame extraction decoding on the target video stream, which can avoid excessive occupation of a single processor resource, achieve processor load balancing, and thus improve the processing efficiency of the video stream detection task.

[0113] Figure 6 A schematic diagram of the hardware structure of an electronic device that implements the embodiments of the present application is shown. Referring to this figure, at the hardware level, electronic device 600 includes a processor 610, and optionally includes an internal bus 620, a network interface 630, and a memory 640. The memory 640 may include a memory 641, such as a high-speed random-access memory (RAM), and may also include a non-volatile memory 642, such as at least one disk storage device. Of course, the electronic device may also include hardware required for other services.

[0114] The processor 610, network interface 630, and memory can be interconnected via an internal bus 620. This internal bus 620 can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus. Such buses can be classified as address buses, data buses, control buses, and the like. For ease of illustration, only one bidirectional arrow is used in this figure, but this does not imply that there is only one bus or only one type of bus.

[0115] The memory 640 stores programs. Specifically, the programs may include program codes, which include computer operating instructions. The memory 640 may include a memory 641 and a non-volatile memory 642, and provides instructions and data to the processor 610.

[0116] The processor 610 reads the corresponding computer program from the non-volatile memory 642 into the memory and then runs it, forming a device for locating the target user at the logical level. The processor 610 executes the program stored in the memory and specifically performs the following: Figure 1 or Figure 4 The methods disclosed in the illustrated embodiments implement the functions and beneficial effects of the various methods described in the foregoing method embodiments, which will not be described in detail here.

[0117] The above application Figure 1 or Figure 4 The methods disclosed in the illustrated embodiments can be applied to or implemented by processor 610. Processor 610 may be an integrated circuit chip with signal processing capabilities. During implementation, each step of the above method can be performed by hardware integrated logic circuits or software instructions within processor 610. The processor 610 may be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it may also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. The methods, steps, and logic block diagrams disclosed in the embodiments of this application can be implemented or executed. The general-purpose processor may be a microprocessor or any conventional processor. The steps of the methods disclosed in the embodiments of this application can be directly executed by a hardware decoding processor or by a combination of hardware and software modules within the decoding processor. The software module can be located in a storage medium well-known in the art, such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, etc. The storage medium is located in the memory, and the processor reads the information in the memory and, in conjunction with its hardware, completes the steps of the above method.

[0118] The computer device can also execute the methods described in the above method embodiments and realize the functions and beneficial effects of the methods described in the above method embodiments, which will not be repeated here.

[0119] Of course, in addition to software implementation, the electronic device of this application does not exclude other implementation methods, such as logic devices or a combination of software and hardware, etc. That is to say, the execution subject of the following processing flow is not limited to each logic unit, but can also be hardware or logic devices.

[0120] The embodiment of the present application further provides a computer-readable storage medium, wherein the computer-readable storage medium stores one or more programs, which, when executed by an electronic device including multiple application programs, enables the electronic device to execute Figure 1 or Figure 4 The methods disclosed in the illustrated embodiments implement the functions and beneficial effects of the various methods described in the foregoing method embodiments, which will not be described in detail here.

[0121] The computer-readable storage medium includes a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0122] Furthermore, an embodiment of the present application provides a computer program product, comprising a computer program stored on a non-transitory computer-readable storage medium, wherein the computer program comprises program instructions. When the program instructions are executed by a computer, the following process is implemented: Figure 1 or Figure 4 The methods disclosed in the illustrated embodiments implement the functions and beneficial effects of the various methods described in the foregoing method embodiments, which will not be described in detail here.

[0123] The embodiments of the present application can be applied to various electronic device collaboration or interconnection scenarios, including: collaboration and interconnection between mobile phones and laptops / tablets; collaboration and interconnection between mobile terminals and smart TVs / displays; collaboration and interconnection between mobile phones or tablets and in-car entertainment systems; collaboration and interconnection between mobile terminals and smart conference systems, etc., thereby meeting the diverse needs of users in scenarios such as smart homes, smart offices, and smart travel.

[0124] In short, the above description is only a preferred embodiment of the present application and does not limit the scope of protection of the present application. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present application shall be included in the scope of protection of the present application.

[0125] The systems, devices, modules, or units described in the above embodiments may be implemented by computer chips or entities, or by products having certain functions. A typical implementation device is a computer. Specifically, the computer may be, for example, a personal computer, a laptop computer, a cellular phone, a camera phone, a smartphone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, a wearable device, or a combination of any of these devices.

[0126] Computer-readable media includes permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. The information can be computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassettes, magnetic disk storage or other magnetic storage devices, or any other non-transmission media that can store information that can be accessed by a computing device. As defined herein, computer-readable media does not include transitory computer-readable media such as modulated data signals and carrier waves.

[0127] It should also be noted that the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, commodity, or apparatus that includes a series of elements includes not only those elements but also other elements not explicitly listed, or includes elements inherent to such process, method, commodity, or apparatus. In the absence of further limitations, an element defined by the phrase "comprises a ..." does not exclude the presence of other identical elements in the process, method, commodity, or apparatus that includes the element.

[0128] The various embodiments in this specification are described in a progressive manner. Similar parts between the various embodiments can be referred to in conjunction with each other. Each embodiment focuses on the differences between the other embodiments. In particular, the system embodiments are generally similar to the method embodiments, so the description is relatively simple. For relevant parts, refer to the description of the method embodiments.

Claims

1. A video stream detection method, characterized in that: include: Determine the frame extraction frequency based on the event detection task of the target video stream; Determining, according to the frame extraction frequency, one of a graphics processing unit (GPU) and a central processing unit (CPU) as a first processor; Performing frame extraction decoding on the target video stream by the first processor to obtain a video decoding stream; The event detection task is performed on the video decoding stream to obtain an event detection result.

2. The method according to claim 1, characterized in that The step of determining, based on the frame extraction frequency, one of a graphics processing unit (GPU) and a central processing unit (CPU) as the first processor includes: When the frame extraction frequency is greater than a preset first frequency threshold, determining that the GPU is the first processor; When the frame extraction frequency is less than or equal to the first frequency threshold, the CPU is determined to be the first processor.

3. The method according to claim 1, characterized in that The step of performing frame extraction decoding on the target video stream by the first processor to obtain a decoded video stream includes: performing frame extraction and decoding processing on the target video stream by the first processor, and monitoring resource utilization of the first processor during the frame extraction and decoding processing; When the resource utilization rate of the first processor reaches a preset threshold, determining one of the GPU and the CPU as the second processor according to a first load of the CPU and a second load of the GPU; The target video stream is switched to the second processor for frame extraction and decoding processing to obtain a video decoding stream.

4. The method according to claim 3, characterized in that After determining, based on the first load of the CPU and the second load of the GPU, that one of the GPU and the CPU is the second processor, the method further includes: When the first load is greater than the second load, the frame extraction frequencies of multiple tasks in the CPU are obtained, and the target tasks whose frame extraction frequencies are greater than a preset second frequency threshold among the multiple tasks are transferred to the GPU for frame extraction decoding processing.

5. The method according to claim 1, wherein The step of performing frame extraction decoding on the target video stream by the first processor to obtain a decoded video stream includes: Performing frame extraction decoding on the target video stream by the first processor to obtain an initial video decoding stream; Determining the similarity between any two adjacent video frames in the initial video decoded stream; When the similarity is greater than a preset similarity threshold, the target video frame in any two adjacent video frames is filtered out to obtain a video decoding stream.

6. The method according to any one of claims 1 to 5, characterized in that The performing the event detection task on the video decoding stream to obtain an event detection result includes: Determining, according to the event type of the event detection task, a prompt word corresponding to the event type; The video decoding stream and the prompt word are input into a pre-trained language model, and the video decoding stream is detected according to the prompt word by the pre-trained language model to obtain an event detection result.

7. A video stream detection device, characterized in that: include: A frame extraction frequency determination module is used to determine the frame extraction frequency according to the event detection task of the target video stream; a processor determining module, configured to determine, based on the frame extraction frequency, one of a graphics processing unit (GPU) and a central processing unit (CPU) as a first processor; a frame extraction decoding processing module, configured to perform frame extraction decoding processing on the target video stream through the first processor to obtain a video decoding stream; The detection result determination module is used to perform the event detection task on the video decoding stream to obtain an event detection result.

8. An electronic device, characterized in that: The electronic device includes a processor and a memory, wherein the memory stores programs or instructions that can be run on the processor, and when the programs or instructions are executed by the processor, the steps of the method according to any one of claims 1 to 6 are implemented.

9. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a program or instruction, and when the program or instruction is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

10. A computer program product, characterized in that The computer program product comprises a computer program stored on a non-transitory computer-readable storage medium, wherein the computer program comprises program instructions, which, when executed by a computer, cause the computer to perform the steps of the method according to any one of claims 1 to 6.

Images