Video frame extraction method, system and apparatus

Abstract

The application discloses a video frame extraction method, system and device. The method comprises the following steps: acquiring shared sample information published by a monitoring point owner; receiving a streaming media distribution request of a target monitoring point indicated by the shared sample information, and performing video stream distribution on a media stream corresponding to the streaming media distribution request; detecting whether to perform frame extraction processing on the video stream based on a predetermined condition, wherein the predetermined condition comprises whether the target monitoring point has a shared sample permission and whether there is an artificial intelligence algorithm analysis requirement; and executing a corresponding strategy according to a detection result, wherein the strategy is used at least for indicating whether to perform frame extraction and whether to input sample picture data obtained through frame extraction processing to a sample management network element. The application solves the technical problems of low processing video data efficiency and less artificial intelligence algorithm sample data caused by the fact that a front video frame extraction link occupies most of the analysis time of an intelligent monitoring platform.

Description

Technical Field

[0001] This application relates to the field of video surveillance, and more specifically, to a video frame extraction method, system, and apparatus. Background Technology

[0002] Deep learning is essentially an artificial neural network. It's a mimicry of the human brain, imitating human learning abilities by simulating the brain's biological neural networks. Deep learning learns the inherent patterns and hierarchical representations of sample data; it's a complex machine learning algorithm that discovers the distributed feature representations of data.

[0003] Deep learning is a type of machine learning, and machine learning is an essential path to achieving artificial intelligence (AI). Besides setting up a large number of neural network nodes and designing the model, the most crucial aspect of deep learning is that it must be trained with sample data to enable the machine to acquire certain capabilities. The more samples used for this training, the higher the accuracy.

[0004] Monitoring platforms have evolved from simple video surveillance to intelligent monitoring platforms that combine monitoring and intelligent analysis. However, the application of artificial intelligence algorithms in the field of intelligent monitoring faces the following common problems:

[0005] (1) The pre-video frame extraction process occupies most of the overall service time of the intelligent monitoring platform, resulting in low efficiency in processing video data; (2) In the research and development process, the accuracy improvement of deep learning type artificial intelligence algorithms depends on a large number of samples, so how to obtain a large number of samples to train artificial intelligence algorithms is also one of the problems; (3) The operator's monitoring platform has rich monitoring point resources, and how to share monitoring point resources reasonably and safely presents a problem at the level of resource management.

[0006] There is currently no effective solution to the above problems. Summary of the Invention

[0007] This application provides a video frame extraction method, system, and apparatus to at least solve the technical problems of low efficiency in processing video data and insufficient sample data for artificial intelligence algorithms caused by the pre-processing video frame extraction occupying most of the analysis time of the intelligent monitoring platform.

[0008] According to one aspect of the embodiments of this application, a video frame extraction method is provided, comprising: acquiring shared sample information published by the owner of a monitoring point; receiving a streaming media distribution request from a target monitoring point indicated by the shared sample information, and distributing the media stream corresponding to the streaming media distribution request as a video stream; detecting whether to perform frame extraction processing on the video stream based on predetermined conditions, wherein the predetermined conditions include: whether the target monitoring point has shared sample permissions and whether there is a need for artificial intelligence algorithm analysis, wherein the need for artificial intelligence algorithm analysis is used to indicate whether the image data corresponding to the streaming media needs to be analyzed based on an artificial intelligence algorithm model; and executing a corresponding strategy according to the detection result, wherein the strategy is at least used to indicate whether to perform frame extraction and whether to input the sample image data obtained from the frame extraction processing to a sample management network element.

[0009] Optionally, after obtaining the target sample image data, the method further includes: if the detection result indicates that the monitoring point has the right to share the sample, extracting frames from the video stream to obtain sample image data, and inputting the sample image data into the sample management network element, wherein the data stored in the sample management network element has the right to share the sample; if the detection result indicates that the monitoring point only has the need for artificial intelligence algorithm analysis, extracting frames from the video stream to obtain sample image data, and inputting the sample image data into the artificial intelligence algorithm application module.

[0010] Optionally, inputting sample image data into a sample management network element includes: storing image data in a sample pool within the sample management network element; and labeling the images in the sample pool to obtain labeled images, wherein the labeled images are used as a source of sample images for training artificial intelligence algorithms.

[0011] Optionally, corresponding strategies can be implemented based on the detection results, including: if the detection results indicate that the monitoring point has no sharing permissions and there is no need for artificial intelligence algorithm analysis, refusing to perform frame extraction on the video stream and distributing the video stream to the monitoring application.

[0012] Optionally, frame extraction of the media stream includes: performing a stream fetching operation on the media stream queue; decoding the fetched media stream and extracting keyframes from the video stream; encoding the keyframes and encapsulating them into images.

[0013] Optionally, the retrieved media stream is decoded to extract keyframes, and the keyframes are encoded and encapsulated into images. This includes: calling the GPU processor to decode the video stream and obtain keyframes from the video stream; calling the CPU processor to encode the keyframes and encapsulate the encoded keyframes into images; or selecting a processor from the GPU processor and the CPU processor for extracting keyframes and for encoding and encapsulating keyframes based on a first GPU processor utilization rate and a second CPU processor utilization rate.

[0014] Optionally, selecting a processor for extracting keyframes and a processor for encoding and encapsulating keyframes from the GPU processor and CPU processor based on the first utilization rate of the GPU processor and the second utilization rate of the CPU processor includes: extracting keyframes based on the GPU processor and encoding and encapsulating keyframes based on the CPU processor when the first utilization rate and the second utilization rate are the same; determining the difference between the first utilization rate and the second utilization rate; extracting keyframes based on the CPU processor and encoding and encapsulating keyframes based on the GPU processor when the difference is less than a first preset threshold; and extracting keyframes based on the CPU processor and encoding and encapsulating keyframes when the difference is greater than a second preset threshold, wherein the first preset threshold is less than the second preset threshold.

[0015] According to another aspect of the embodiments of this application, a system for a video frame extraction method is also provided, comprising: a streaming media distribution network element, the streaming media distribution network element comprising: a distribution management logic module and a frame extraction module, wherein the distribution management logic module is used to detect whether the video stream from the monitoring point has a frame extraction requirement, and the frame extraction module is used to extract frames from the video stream to obtain sample image data when the video stream has a frame extraction requirement, wherein the frame extraction requirement is used to indicate whether the monitoring point has shared sample permissions and whether there is a need for artificial intelligence algorithm analysis; a sample management module is used at least to classify and manage the sample images according to the shared sample permissions of the monitoring point.

[0016] Optionally, the frame extraction module is used to call the GPU processor to decode the video stream and obtain keyframes in the video stream; call the CPU processor to encode the keyframes and encapsulate the encoded keyframes into images.

[0017] Optionally, the frame extraction module is also used to balance the task processes of the GPU processor and the CPU processor according to the utilization rate of the GPU processor and the CPU processor, and to evenly distribute the task processes corresponding to frame extraction to the GPU processor and the CPU processor to extract sample images.

[0018] According to another aspect of the embodiments of this application, a video frame extraction device is also provided, comprising: an acquisition module, configured to acquire shared sample information published by the owner of a monitoring point; a distribution module, configured to receive a streaming media distribution request from a target monitoring point indicated by the shared sample information, and to distribute the media stream corresponding to the streaming media distribution request as a video stream; a frame extraction module, configured to detect whether to perform frame extraction processing on the video stream based on predetermined conditions, wherein the predetermined conditions include: whether the target monitoring point has shared sample permissions and whether there is a need for artificial intelligence algorithm analysis, wherein the need for artificial intelligence algorithm analysis is used to indicate whether the image data corresponding to the streaming media needs to be analyzed based on an artificial intelligence algorithm model; and an execution module, configured to execute a corresponding strategy according to the detection result, wherein the strategy is at least used to indicate whether to perform frame extraction and whether to input the sample image data obtained from the frame extraction processing to a sample management network element.

[0019] According to another aspect of the embodiments of this application, a non-volatile storage medium is also provided, comprising: the storage medium including a stored program, wherein, when the program is running, it controls the device where the storage medium is located to execute any video frame extraction method.

[0020] According to another aspect of the embodiments of this application, an electronic device is also provided, including: a processor; a memory for storing processor-executable instructions; wherein the processor is configured to execute instructions to implement any video frame extraction method.

[0021] In this embodiment, a method is adopted whereby the distribution management logic module determines whether a sample has sharing permissions and whether there is an AI requirement, and the frame extraction module extracts frames from video streams with sharing permissions or AI requirements. This is achieved by obtaining shared sample information published by the monitoring point owner; receiving streaming media distribution requests from target monitoring points indicated by the shared sample information; distributing video streams corresponding to the streaming media distribution requests; detecting whether to perform frame extraction processing on the video stream based on predetermined conditions, including whether the target monitoring point has sharing sample permissions and whether there is an artificial intelligence algorithm analysis requirement; and executing corresponding strategies based on the detection results. The strategies are used to indicate whether to perform frame extraction and whether to input the sample image data obtained from frame extraction processing into the sample management network element. This achieves the purpose of improving server performance, synchronizing video distribution and video frame extraction, and enabling GPU and CPU processors to cooperate in processing frame extraction. This improves the efficiency of the frame extraction module, reduces frame extraction latency, and reduces network bandwidth during operation, thereby reducing server performance consumption. It also allows for the acquisition of a large amount of shared sample data for training artificial intelligence algorithms, thus solving the technical problems of low efficiency in processing video data and insufficient sample data for artificial intelligence algorithms caused by the pre-processing video frame extraction occupying most of the analysis time of the intelligent monitoring platform. Attached Figure Description

[0022] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments of this application and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings:

[0023] Figure 1 This is a flowchart illustrating a video frame extraction method according to an embodiment of this application;

[0024] Figure 2 This is a schematic diagram of a system architecture according to an embodiment of this application;

[0025] Figure 3 This is a schematic diagram of an efficient frame extraction module according to an embodiment of this application;

[0026] Figure 4 This is a schematic flowchart of a system flowchart according to an embodiment of this application;

[0027] Figure 5 This is a flowchart illustrating a frame extraction module according to an embodiment of this application;

[0028] Figure 6 This is a schematic diagram of the apparatus structure for a video frame extraction method according to an embodiment of this application;

[0029] Figure 7 A schematic block diagram of an example electronic device 700 that can be used to implement embodiments of this application is shown. Detailed Implementation

[0030] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort should fall within the scope of protection of the present application.

[0031] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0032] According to an embodiment of this application, a video frame extraction method embodiment is provided. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Furthermore, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0033] Figure 1 This is a video frame extraction method according to an embodiment of this application, such as... Figure 1 As shown, the method includes the following steps:

[0034] Step S102: Obtain the shared sample information published by the monitoring point owner;

[0035] It should be noted that shared sample information refers to the monitoring points of shared samples.

[0036] Step S104: Receive the streaming media distribution request of the target monitoring point indicated by the shared sample information, and distribute the media stream corresponding to the streaming media distribution request as a video stream;

[0037] Step S106: Detect whether to perform frame extraction processing on the video stream based on predetermined conditions. The predetermined conditions include: whether the target monitoring point has the permission to share samples and whether there is a need for artificial intelligence algorithm analysis. The need for artificial intelligence algorithm analysis is used to indicate whether the image data corresponding to the streaming media needs to be analyzed based on the artificial intelligence algorithm model.

[0038] Step S108: Execute the corresponding strategy based on the detection results. The strategy is at least used to indicate whether to perform frame extraction and whether to input the sample image data obtained from frame extraction to the sample management network element.

[0039] Understandably, the strategy can also be used to indicate whether to input the sample image data obtained from frame extraction into the artificial intelligence algorithm application module.

[0040] In this embodiment, a method is adopted whereby the distribution management logic module determines whether a sample has sharing permissions and whether there is an AI requirement, and the frame extraction module extracts frames from video streams with sharing permissions or AI requirements. This is achieved by obtaining shared sample information published by the monitoring point owner; receiving streaming media distribution requests from target monitoring points indicated by the shared sample information; distributing video streams corresponding to the streaming media distribution requests; detecting whether to perform frame extraction processing on the video stream based on predetermined conditions, including whether the target monitoring point has sharing sample permissions and whether there is an artificial intelligence algorithm analysis requirement; and executing corresponding strategies based on the detection results. The strategies are used to indicate whether to perform frame extraction and whether to input the sample image data obtained from frame extraction processing into the sample management network element. This achieves the purpose of improving server performance, synchronizing video distribution and video frame extraction, and enabling GPU and CPU processors to cooperate in processing frame extraction. This improves the efficiency of the frame extraction module, reduces frame extraction latency, and reduces network bandwidth during operation, thereby reducing server performance consumption. It also allows for the acquisition of a large amount of shared sample data for training artificial intelligence algorithms, thus solving the technical problems of low efficiency in processing video data and insufficient sample data for artificial intelligence algorithms caused by the pre-processing video frame extraction occupying most of the analysis time of the intelligent monitoring platform.

[0041] In an exemplary embodiment of this application, after obtaining the target sample image data, the method further includes: if the detection result indicates that the monitoring point has the right to share the sample, extracting frames from the video stream to obtain sample image data, and inputting the sample image data into the sample management network element, wherein the data stored in the sample management network element has the right to share the sample; if the detection result indicates that the monitoring point only has the need for artificial intelligence algorithm analysis, extracting frames from the video stream to obtain sample image data, and inputting the sample image data into the artificial intelligence algorithm application module.

[0042] Optionally, inputting sample image data into a sample management network element includes: storing image data in a sample pool within the sample management network element; and labeling the images in the sample pool to obtain labeled images, wherein the labeled images are used as a source of sample images for training artificial intelligence algorithms.

[0043] It should be noted that annotation methods include, but are not limited to, manual annotation or machine-assisted annotation.

[0044] In some optional embodiments of this application, a corresponding strategy is executed based on the detection result, including: if the detection result indicates that the monitoring point has no sharing permission and there is no need for artificial intelligence algorithm analysis, refusing to perform frame extraction operation on the video stream and distributing the video stream to the monitoring application.

[0045] It should be noted that monitoring applications include, but are not limited to, remote user access, AI algorithm configuration, and platform storage.

[0046] As an optional implementation, frame extraction of the media stream includes: performing a stream fetching operation on the media stream queue; decoding the fetched media stream and extracting keyframes from the video stream; encoding the keyframes and encapsulating them into images.

[0047] It is understandable that the frame extraction module has two stages. The first stage is to decode the video stream after it is pulled and extract the key frames of the video stream. The second stage is to encode the key frames and encapsulate them into images.

[0048] In an exemplary embodiment of this application, the process of decoding the retrieved media stream and extracting keyframes from the video stream, encoding the keyframes, and encapsulating them into images includes: calling a GPU processor to decode the video stream and obtain keyframes from the video stream; calling a CPU processor to encode the keyframes and encapsulate the encoded keyframes into images; or selecting a processor for extracting keyframes and a processor for encoding and encapsulating keyframes from the GPU processor and the CPU processor based on a first utilization rate of the GPU processor and a second utilization rate of the CPU processor.

[0049] It should be noted that GPU processors are more efficient at video decoding, so the first step in the frame extraction module prioritizes the use of GPU processors for processing.

[0050] Optionally, selecting a processor for extracting keyframes and a processor for encoding and encapsulating keyframes from the GPU processor and CPU processor based on the first utilization rate of the GPU processor and the second utilization rate of the CPU processor includes: extracting keyframes based on the GPU processor and encoding and encapsulating keyframes based on the CPU processor when the first utilization rate and the second utilization rate are the same; determining the difference between the first utilization rate and the second utilization rate; extracting keyframes based on the CPU processor and encoding and encapsulating keyframes based on the GPU processor when the difference is less than a first preset threshold; and extracting keyframes based on the CPU processor and encoding and encapsulating keyframes when the difference is greater than a second preset threshold, wherein the first preset threshold is less than the second preset threshold.

[0051] Understandably, when the difference between the first occupancy rate and the second occupancy rate is less than the first preset threshold, that is, the difference between the GPU processor and the CPU processor occupancy rate is small, keyframes can be extracted based on the CPU processor and encoded and encapsulated based on the GPU processor; when the difference between the first occupancy rate and the second occupancy rate is greater than the second preset threshold, that is, the difference between the GPU processor and the CPU processor occupancy rate is large, and the CPU processor occupancy rate is relatively lower than the GPU processor, keyframes can be extracted based on the CPU processor and encoded and encapsulated, wherein the first preset threshold is less than the second preset threshold.

[0052] For example, when the number of processing processes of the GPU processor is 4 and the number of processing processes of the CPU processor is 4, in order to balance the number of processes of the GPU processor and the CPU processor, the above frame extraction process is evenly distributed to the GPU processor and the CPU processor. That is, the processing process of decoding and extracting key frames can be assigned to the GPU processor, and the processing process of encoding and encapsulation can be assigned to the CPU processor, instead of using the GPU processor or the CPU processor alone, so that the GPU processor and the CPU processor can process processes efficiently.

[0053] Figure 2 This is a schematic diagram of a system architecture according to an embodiment of this application, such as... Figure 2 As shown, the system architecture mainly includes the following structures:

[0054] The streaming media distribution network element includes a distribution management logic module and a frame extraction module. The distribution management logic module is used to detect whether the video stream from the monitoring point has a frame extraction requirement. The frame extraction module is used to extract frames from the video stream to obtain sample image data when the video stream has a frame extraction requirement. The frame extraction requirement is used to indicate whether the monitoring point has the permission to share samples and whether there is a need for artificial intelligence algorithm analysis.

[0055] The sample management module is used at least to classify and manage sample images based on the shared sample permissions of the monitoring points.

[0056] It should be noted that the system also includes the following structure: device management network element, used to publish monitoring points for shared samples; and artificial intelligence algorithm application, used to analyze the media stream after frame extraction.

[0057] In some optional embodiments of this application, the frame extraction module is used to call the GPU processor to decode the video stream and obtain key frames in the video stream; call the CPU processor to encode the key frames and encapsulate the encoded key frames into images.

[0058] As an optional implementation, the frame extraction module is also used to balance the task processes of the GPU processor and the CPU processor according to the utilization rate of the GPU processor and the CPU processor, and to evenly distribute the task processes corresponding to frame extraction to the GPU processor and the CPU processor to extract sample images.

[0059] Figure 3 This is a schematic diagram of an efficient frame-skipping module according to an embodiment of this application, such as... Figure 3 As shown, the module diagram includes the following structure:

[0060] The keyframe extraction module is used to decode the retrieved media stream and extract keyframes from the video stream. The retrieved media stream is obtained by pulling the stream from the media stream queue.

[0061] The encoding and encapsulation module is used to encode keyframes and encapsulate them into images.

[0062] It should be noted that the frame extraction module and the encoding and packaging module are processed by the GPU processor and the CPU processor in cooperation to handle the frame extraction tasks of the above modules.

[0063] To facilitate a better understanding of the technical solutions of this application by those skilled in the art, a specific embodiment will now be described.

[0064] Figure 4 This is a schematic flowchart of a system flowchart according to an embodiment of this application, such as... Figure 4 As shown, the process mainly includes the following steps:

[0065] (1) The owner of the monitoring point has the authority to publish shared samples;

[0066] (2) The distribution management logic module that synchronizes the published information to the streaming media distribution network element;

[0067] (3) When a monitoring point has a streaming media distribution request, the video stream needs to be distributed through the streaming media distribution network element;

[0068] It should be noted that the streaming media distribution request is a distribution request for the corresponding monitoring application, which includes, but is not limited to, remote user access, artificial intelligence algorithm configuration, and platform storage.

[0069] (4) Monitor the front-end media stream to the video distribution module;

[0070] (5) The distribution management logic module determines the frame extraction requirement;

[0071] It should be noted that if there is no shared sample permission and no AI requirement, frame extraction is not performed; the video stream is simply distributed to the monitoring application, and the process ends. If there is shared sample permission or AI requirement, the video stream simultaneously enters the efficient frame extraction module.

[0072] (6) The high-efficiency frame extraction module extracts frames to form images;

[0073] (7) Determine whether the monitoring point has permission to share samples;

[0074] It should be noted that when there is only a need for artificial intelligence algorithm application, the image enters the artificial intelligence algorithm application module and the process ends; when there is sample sharing permission, the image generated by frame extraction enters the sample management network element.

[0075] (8) Sample labeling and management are carried out with the assistance of humans or machines;

[0076] (9) The labeled samples serve as the source of training for artificial intelligence algorithms;

[0077] It is noteworthy that the frame extraction module of this application makes full use of the cooperation between the GPU processor and the CPU processor. In the process of extracting keyframes from the video stream and encoding and encapsulating the keyframes into images, the GPU processor and the CPU processor cooperate to process the frame extraction process, which maximizes the performance of the server, improves the efficiency of the frame extraction module, and reduces the frame extraction latency.

[0078] This application proposes a management mechanism for the publishing permissions of monitoring point owners, which achieves the goal of using the monitoring resources of the monitoring platform in a reasonable and secure manner. It extracts frames from the monitoring resources of the monitoring platform as a source of training samples for artificial intelligence algorithms, thereby realizing the secondary operation of the monitoring platform's resources.

[0079] In terms of architecture design, this application deploys the frame extraction module in the streaming media distribution network element. It does not need to specifically pull video streams from users. Instead, when users have remote monitoring or AI application needs and require the streaming media network element to distribute video, frame extraction is performed synchronously, which reduces the network bandwidth and performance consumption of the streaming media network element pulling the stream centrally.

[0080] This application proposes a system and method for the reasonable and secure use of these resources, enabling monitoring points, with the owner's authorization, to become a source of training samples for artificial intelligence algorithms. The preliminary step in generating artificial intelligence algorithms and samples is frame extraction, which is resource-intensive and time-consuming. This application proposes to comprehensively utilize GPU and CPU processors for frame extraction and the encoding and encapsulation of keyframes into samples, improving overall server efficiency, reducing latency, and making it more practical in real-world applications.

[0081] Figure 5This is a flowchart illustrating a frame extraction module according to an embodiment of this application, as shown below. Figure 5 As shown, the flowchart includes the following steps:

[0082] (1) Perform a stream fetch operation on the queue of the media stream;

[0083] (2) Decode the video stream after it is pulled and extract the key frames of the video stream;

[0084] (3) Encode the keyframes and encapsulate them into images.

[0085] It should be noted that when decoding the video stream and extracting keyframes from the retrieved media stream, the GPU processor can be used to process the video stream, encode the keyframes and encapsulate them into images, or the CPU processor can be used to process the keyframes; or the task processes of the GPU processor and CPU processor can be balanced according to the current system's GPU processor and CPU processor utilization.

[0086] For example, when the GPU processor has 4 processing processes and the CPU processor has 4 processing processes, in order to balance the number of processes on the GPU and CPU processors, the above-mentioned frame extraction process is evenly distributed between the GPU and CPU processors. That is, the decoding and keyframe extraction process can be assigned to the GPU processor, and the encoding and encapsulation process can be assigned to the CPU processor, instead of using only the GPU processor or the CPU processor. This allows the GPU and CPU processors to process processes efficiently. For example, when the GPU processor has 3 processing processes and the CPU processor has 7 processing processes, in order to balance the number of processes on the GPU and CPU processors, the decoding and keyframe extraction process and the encoding and encapsulation process can be assigned to the GPU processor.

[0087] Figure 6 This is a schematic diagram of the apparatus structure for a video frame extraction method according to an embodiment of this application, as shown below. Figure 6 As shown, the device includes:

[0088] Module 60 is used to acquire shared sample information published by the owner of the monitoring point;

[0089] Distribution module 62 is used to receive streaming media distribution requests from target monitoring points indicated by shared sample information, and to distribute the media streams corresponding to the streaming media distribution requests as video streams;

[0090] The frame extraction module 64 is used to detect whether to perform frame extraction processing on the video stream based on predetermined conditions. The predetermined conditions include: whether the target monitoring point has the permission to share samples and whether there is a need for artificial intelligence algorithm analysis. The need for artificial intelligence algorithm analysis is used to indicate whether the image data corresponding to the streaming media needs to be analyzed based on the artificial intelligence algorithm model.

[0091] The execution module 68 is used to execute the corresponding strategy based on the detection result. The strategy is at least used to indicate whether to perform frame extraction and whether to input the sample image data obtained by frame extraction to the sample management network element.

[0092] In this device, the acquisition module 60 is used to acquire shared sample information published by the owner of the monitoring point; the distribution module 62 is used to receive the streaming media distribution request from the target monitoring point indicated by the shared sample information, and distribute the media stream corresponding to the streaming media distribution request as a video stream; the frame extraction module 64 is used to detect whether to perform frame extraction processing on the video stream based on predetermined conditions, wherein the predetermined conditions include: whether the target monitoring point has the right to share samples and whether there is a need for artificial intelligence algorithm analysis, wherein the need for artificial intelligence algorithm analysis is used to indicate whether the image data corresponding to the streaming media needs to be analyzed based on an artificial intelligence algorithm model; and the execution module 68 is used to execute the corresponding strategy according to the detection results. The strategy is used to indicate whether to perform frame extraction and whether to input the sample image data obtained from frame extraction to the sample management network element. This achieves the goals of improving server performance, synchronizing video distribution and video frame extraction, and enabling GPU and CPU processors to cooperate in processing frame extraction. This improves the efficiency of the frame extraction module, reduces frame extraction latency, and reduces network bandwidth during operation, thereby reducing server performance consumption. It also allows for the acquisition of a large amount of shared sample data for training artificial intelligence algorithms. This solves the technical problems of low efficiency in processing video data and insufficient sample data for artificial intelligence algorithms caused by the pre-processing video frame extraction occupying most of the analysis time of the intelligent monitoring platform.

[0093] According to another aspect of the embodiments of this application, a non-volatile storage medium is also provided, the non-volatile storage medium including a stored program, wherein, when the program is running, it controls the device where the non-volatile storage medium is located to execute any video frame extraction method.

[0094] Specifically, the aforementioned storage medium is used to store program instructions for the following functions, thereby implementing the following functions:

[0095] The system acquires shared sample information published by the owner of the monitoring point; receives streaming media distribution requests from target monitoring points indicated by the shared sample information, and distributes the media streams corresponding to the streaming media distribution requests as video streams; detects whether to perform frame extraction processing on the video stream based on predetermined conditions, wherein the predetermined conditions include: whether the target monitoring point has shared sample permissions and whether there is a need for artificial intelligence algorithm analysis; and executes corresponding strategies based on the detection results, wherein the strategies are at least used to indicate whether to perform frame extraction and whether to input the sample image data obtained from frame extraction processing into the sample management network element.

[0096] Optionally, in this embodiment, the storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or equipment, or any suitable combination of the foregoing. More specific examples of the storage medium include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0097] In an exemplary embodiment of this application, a computer program product is also provided, including a computer program that, when executed by a processor, implements any of the above-described video frame extraction methods.

[0098] Optionally, when executed by a processor, the computer program may perform the following steps:

[0099] The system acquires shared sample information published by the owner of the monitoring point; receives streaming media distribution requests from target monitoring points indicated by the shared sample information, and distributes the media streams corresponding to the streaming media distribution requests as video streams; detects whether to perform frame extraction processing on the video stream based on predetermined conditions, wherein the predetermined conditions include: whether the target monitoring point has shared sample permissions and whether there is a need for artificial intelligence algorithm analysis; and executes corresponding strategies based on the detection results, wherein the strategies are at least used to indicate whether to perform frame extraction and whether to input the sample image data obtained from frame extraction processing into the sample management network element.

[0100] An electronic device is provided according to an embodiment of this application, the electronic device including: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform any of the above-described video frame extraction methods.

[0101] Optionally, the electronic device may further include a transmission device and an input / output device, wherein the transmission device is connected to the processor, and the input / output device is connected to the processor.

[0102] Figure 7 A schematic block diagram of an example electronic device 700 that can be used to implement embodiments of this application is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the application described and / or claimed herein.

[0103] like Figure 7 As shown, device 700 includes a computing unit 701, which can perform various appropriate actions and processes based on a computer program stored in read-only memory (ROM) 702 or a computer program loaded from storage unit 708 into random access memory (RAM) 703. RAM 703 may also store various programs and data required for the operation of device 700. The computing unit 701, ROM 702, and RAM 703 are interconnected via bus 704. Input / output (I / O) interface 705 is also connected to bus 704.

[0104] Multiple components in device 700 are connected to I / O interface 705, including: input unit 706, such as keyboard, mouse, etc.; output unit 707, such as various types of monitors, speakers, etc.; storage unit 708, such as disk, optical disk, etc.; and communication unit 709, such as network card, modem, wireless transceiver, etc. Communication unit 709 allows device 700 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0105] The computing unit 701 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 701 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 701 performs the various methods and processes described above, such as video frame extraction methods. For example, in some embodiments, the video frame extraction method may be implemented as a computer software program tangibly contained in a machine-readable medium, such as storage unit 708. In some embodiments, part or all of the computer program may be loaded and / or installed on device 700 via ROM 702 and / or communication unit 709. When the computer program is loaded into RAM 703 and executed by the computing unit 701, one or more steps of the video frame extraction method described above may be performed. Alternatively, in other embodiments, the computing unit 701 may be configured to perform the video frame extraction method by any other suitable means (e.g., by means of firmware).

[0106] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0107] The program code used to implement the methods of this application may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing device, such that when executed by the processor or controller, the functions / operations specified in the flowcharts and / or block diagrams are implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0108] In the context of this application, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. Machine-readable media can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0109] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0110] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as a data server), or computing systems that include middleware components (e.g., an application server), or computing systems that include frontend components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with embodiments of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., a communication network). Examples of communication networks include local area networks (LANs), wide area networks (WANs), and the Internet.

[0111] Computer systems can include clients and servers. Clients and servers are generally located far apart and typically interact via communication networks. Client-server relationships are created by computer programs running on the respective computers and having a client-server relationship with each other. Servers can be cloud servers, servers in distributed systems, or servers incorporating blockchain technology.

[0112] The sequence numbers of the embodiments in this application are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0113] In the above embodiments of this application, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0114] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. The device embodiments described above are merely illustrative; for example, the division of units can be a logical functional division, and in actual implementation, there may be other division methods. For instance, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the displayed or discussed mutual coupling, direct coupling, or communication connection may be through some interfaces; the indirect coupling or communication connection between units or modules may be electrical or other forms.

[0115] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0116] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0117] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as a USB flash drive, read-only memory (ROM), random access memory (RAM), portable hard drive, magnetic disk, or optical disk.

[0118] The above description is only a preferred embodiment of this application. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of this application, and these improvements and modifications should also be considered within the scope of protection of this application.

Claims

1. A video frame extraction method, characterized in that, include: Obtain shared sample information published by the owner of the monitoring point; Receive the streaming media distribution request of the target monitoring point indicated by the shared sample information, and distribute the media stream corresponding to the streaming media distribution request as a video stream; Detecting whether to perform frame extraction processing on the video stream based on predetermined conditions includes: calling the GPU processor to decode the video stream and obtain keyframes in the video stream; calling the CPU processor to encode the keyframes and encapsulate the encoded keyframes into images; or selecting a processor for extracting keyframes and a processor for encoding and encapsulating the keyframes from the GPU processor and the CPU processor according to a first utilization rate of the GPU processor and a second utilization rate of the CPU processor, wherein the predetermined conditions include: whether the target monitoring point has shared sample permissions and whether there is a need for artificial intelligence algorithm analysis, wherein the need for artificial intelligence algorithm analysis is used to indicate whether the image data corresponding to the streaming media needs to be analyzed based on an artificial intelligence algorithm model; The corresponding strategy is executed based on the detection results, wherein the strategy is at least used to indicate whether to perform frame extraction and whether to input the sample image data obtained by frame extraction to the sample management network element.

2. The method according to claim 1, characterized in that, After obtaining the target sample image data, the method further includes: If the detection result indicates that the monitoring point has the right to share samples, the video stream is frame-by-frame extracted to obtain sample image data, and the sample image data is input to the sample management network element, wherein the data stored in the sample management network element has the right to share. If the detection result indicates that the monitoring point only requires artificial intelligence algorithm analysis, the video stream is frame-by-frame extracted to obtain sample image data, and the sample image data is input into the artificial intelligence algorithm application module.

3. The method according to claim 2, characterized in that, Inputting the sample image data into the sample management network element includes: The image data is stored in the sample pool within the sample management network element; The images in the sample pool are labeled to obtain labeled images, which are used as sample images for training artificial intelligence algorithms.

4. The method according to claim 1, characterized in that, Based on the detection results, corresponding strategies are executed, including: If the detection result indicates that the monitoring point has no sharing permissions and there is no need for artificial intelligence algorithm analysis, the frame extraction operation on the video stream is refused, and the video stream is distributed to the monitoring application.

5. The method according to claim 1, characterized in that, Frame extraction of the media stream includes: Perform a stream fetch operation on the queue of the media stream; The retrieved media stream is decoded, and keyframes of the video stream are extracted. The keyframes are then encoded and encapsulated as images.

6. The method according to claim 1, characterized in that, Based on a first utilization rate of the GPU processor and a second utilization rate of the CPU processor, a processor for extracting keyframes and a processor for encoding and encapsulating the keyframes are selected from the GPU processor and the CPU processor, including: When the first occupancy rate and the second occupancy rate are the same, key frames are extracted based on the GPU processor, and the key frames are encoded and encapsulated based on the CPU processor. The difference between the first occupancy rate and the second occupancy rate is determined. If the difference is less than a first preset threshold, key frames are extracted based on the CPU processor, and the key frames are encoded and encapsulated based on the GPU processor. If the difference is greater than the second preset threshold, key frames are extracted based on the CPU processor and the key frames are encoded and encapsulated, wherein the first preset threshold is less than the second preset threshold.

7. A video frame extraction system, characterized in that, include: The streaming media distribution network element includes a distribution management logic module and a frame extraction module. The distribution management logic module is used to detect whether the video stream from the monitoring point has a frame extraction requirement. The frame extraction module is used to extract frames from the video stream to obtain sample image data when the video stream has a frame extraction requirement. The frame extraction requirement is used to indicate whether the monitoring point has the permission to share samples and whether there is a need for artificial intelligence algorithm analysis. The sample management module is at least used to classify and manage the sample images according to the shared sample permissions of the monitoring points; The frame extraction module is further configured to call the GPU processor to decode the video stream and obtain keyframes in the video stream; call the CPU processor to encode the keyframes and encapsulate the encoded keyframes into images; and balance the task processes of the GPU processor and CPU processor according to the utilization rate of the GPU processor and CPU processor, and evenly distribute the task processes corresponding to frame extraction to the GPU processor and CPU processor to extract the sample images.

8. A video frame extraction device, characterized in that, include: The acquisition module is used to acquire shared sample information published by the owner of the monitoring point; The distribution module is used to receive streaming media distribution requests from target monitoring points indicated by the shared sample information, and to distribute the media streams corresponding to the streaming media distribution requests as video streams; A frame extraction module is used to detect whether to perform frame extraction processing on the video stream based on predetermined conditions, including: calling the GPU processor to decode the video stream and obtain keyframes in the video stream; calling the CPU processor to encode the keyframes and encapsulate the encoded keyframes into images; or selecting a processor for extracting keyframes and a processor for encoding and encapsulating the keyframes from the GPU processor and the CPU processor according to a first utilization rate of the GPU processor and a second utilization rate of the CPU processor, wherein the predetermined conditions include: whether the target monitoring point has shared sample permissions and whether there is a need for artificial intelligence algorithm analysis, wherein the need for artificial intelligence algorithm analysis is used to indicate whether the image data corresponding to the streaming media needs to be analyzed based on an artificial intelligence algorithm model; The execution module is used to execute the corresponding strategy based on the detection result, wherein the strategy is at least used to indicate whether to perform frame extraction and whether to input the sample image data obtained by frame extraction to the sample management network element.

9. A non-volatile storage medium, characterized in that, The storage medium includes a stored program, wherein, when the program is executed, it controls the device where the storage medium is located to perform the video frame extraction method according to any one of claims 1 to 6.

10. An electronic device, characterized in that, include: processor; Memory used to store the processor's executable instructions; The processor is configured to execute the instructions to implement the video frame extraction method as described in any one of claims 1 to 6.

Images