Abnormity detection result real-time playing method, device and system based on video stream

Abstract

The invention discloses a video stream-based abnormal detection result real-time playing method, device and system, and the method comprises the steps: accessing an original video stream which comprises a first timestamp when each video frame in the original video stream enters a buffer queue; detecting the original video stream based on an anomaly detection model of at least one service type to obtain an anomaly detection result of the service type; the anomaly detection result comprises a second timestamp when the video frame corresponding to the anomaly detection result enters the cache queue; under the condition that the first timestamp and the second timestamp meet a video stream fusion condition, constructing a supplemental enhancement information frame based on an anomaly detection result, and fusing the supplemental enhancement information frame with the original video stream to obtain a target video stream; and sending the target video stream to a front-end player for playing. According to the invention, the supplemental enhancement frame corresponding to the anomaly detection result and the original video stream can be fused, and real-time playing of the anomaly detection result is realized.

Description

Technical Field

[0001] The present invention relates to the field of video transmission technology, and in particular to a method, device and system for real-time playback of abnormality detection results based on video streams. Background Art

[0002] In current industrial scene applications, there is usually anomaly detection for specific scenes in the surveillance video of the on-site camera, such as detection of not wearing a safety helmet, detection of personnel intrusion, detection of suspicious personnel wandering, detection of flame anomalies, detection of belt deviation, detection of gravel blockage, etc., and the abnormal content and abnormal location information obtained by the detection need to be displayed on the real-time monitoring screen, so that production management personnel can view the real-time images of the abnormalities occurring on site as soon as possible, and make emergency decisions quickly according to the abnormal content and the location information of the abnormality, and perform alarm processing.

[0003] At present, the existing technology is to separate the data transmission channel of the video stream and the intelligent algorithm analysis results. That is, one channel transmits the original video stream of the camera, and another data channel is used to transmit the anomaly detection results. The application side receives the data of these two channels at the same time, and displays them in a fusion on the player to show the real-time picture. However, this method requires the server and the client to add and maintain an additional network link in addition to the video playback channel for transmitting the anomaly detection results. And due to the separation of the video stream and the anomaly detection results, the server and the application need to perform operations such as data source matching and time consistency matching between the video stream and the anomaly detection results. If any link fails, the real-time playback requirements of the anomaly detection results cannot be achieved. These disadvantages increase the complexity and instability of the system. Summary of the invention

[0004] The present invention provides a method, device and system for real-time playback of abnormality detection results based on video stream, so as to realize real-time playback of abnormality detection results.

[0005] According to one aspect of the present invention, a method for real-time playback of abnormality detection results based on video stream is provided, comprising:

[0006] Accessing an original video stream, wherein the original video stream includes a first timestamp of each video frame in the original video stream entering a cache queue;

[0007] Detecting the original video stream based on at least one anomaly detection model of a business type, obtaining an anomaly detection result of the business type; the anomaly detection result includes a second timestamp of a video frame corresponding to the anomaly detection result entering a cache queue;

[0008] When the first timestamp and the second timestamp meet the video stream fusion condition, construct a supplementary enhancement information frame based on the anomaly detection result, and fuse the supplementary enhancement information frame with the original video stream to obtain a target video stream;

[0009] The target video stream is sent to a front-end player for playback.

[0010] According to another aspect of the present invention, a real-time playback system for abnormality detection results based on video streams is provided, comprising: at least one video stream acquisition device, a video stream abnormality detection system and a front-end player; the video stream abnormality detection system comprises a video stream access module, at least one video stream abnormality detection module, a video stream fusion module and a video stream publishing module;

[0011] The video stream access module is used to access the original video stream collected by the video stream acquisition device, and the original video stream includes the first timestamp of each video frame in the original video stream entering the cache queue;

[0012] The video stream anomaly detection module is used to detect the original video stream based on any business type and the anomaly detection model of the business type to obtain an anomaly detection result of the business type; the anomaly detection result includes a second timestamp of the video frame corresponding to the anomaly detection result entering the cache queue;

[0013] The video stream fusion module is used to construct a supplementary enhancement information frame based on the anomaly detection result when the first timestamp and the second timestamp meet the video stream fusion condition, and fuse the supplementary enhancement information frame with the original video stream to obtain a target video stream;

[0014] The video stream publishing module is used to send the target video stream to the front-end player for playing.

[0015] According to another aspect of the present invention, an electronic device is provided, the electronic device comprising:

[0016] at least one processor; and

[0017] a memory communicatively connected to the at least one processor; wherein,

[0018] The memory stores a computer program that can be executed by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the real-time playback method of anomaly detection results based on video streams described in any embodiment of the present invention.

[0019] According to another aspect of the present invention, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the real-time playback method of anomaly detection results based on video streams described in any embodiment of the present invention when executed.

[0020] The technical solution of the embodiment of the present invention is to access the original video stream, the original video stream includes the first timestamp of each video frame in the original video stream entering the cache queue; detect the original video stream based on at least one business type anomaly detection model to obtain the business type anomaly detection result; the anomaly detection result includes the second timestamp of the video frame corresponding to the anomaly detection result entering the cache queue; when the first timestamp and the second timestamp meet the video stream fusion conditions, construct a supplementary enhancement information frame based on the anomaly detection result, fuse the supplementary enhancement information frame with the original video stream to obtain the target video stream; send the target video stream to the front-end player for playback. The supplementary enhancement information frame corresponding to the anomaly detection result can be fused with the original video stream to achieve real-time playback of the anomaly detection result.

[0021] It should be understood that the contents described in this section are not intended to identify the key or important features of the embodiments of the present invention, nor are they intended to limit the scope of the present invention. Other features of the present invention will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0023] Figure 1 It is a flowchart of a method for real-time playback of abnormal detection results based on video streams provided in Embodiment 1 of the present invention;

[0024] Figure 2 is a schematic diagram of the data structure of a supplementary enhancement information frame provided in Embodiment 1 of the present invention;

[0025] Figure 3 1 is a schematic diagram of the structure of a real-time playback system for abnormal detection results based on video streams provided in Embodiment 2 of the present invention;

[0026] Figure 4 It is a structural schematic diagram of an electronic device provided in Embodiment 3 of the present invention. DETAILED DESCRIPTION

[0027] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0028] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0029] Embodiment 1

[0030] Figure 1 This is a flowchart of a method for real-time playback of abnormal detection results based on video streams provided in the first embodiment of the present invention. This embodiment is applicable to the real-time playback of abnormal detection results of monitoring videos of on-site cameras in industrial scenes. The method can be executed by a real-time playback system for abnormal detection results based on video streams. The real-time playback system for abnormal detection results based on video streams can be implemented in the form of hardware and / or software. The real-time playback system for abnormal detection results based on video streams can be configured in electronic devices such as computers and servers. Figure 1 As shown, the method includes:

[0031] S110 . Access an original video stream, where the original video stream includes a first timestamp of each video frame in the original video stream entering a cache queue.

[0032] Among them, the original video stream refers to the video stream of the video stream acquisition device in the industrial scene. In an embodiment of the present invention, the original video stream of at least one video stream acquisition device is accessed through a standard streaming media protocol. Specifically, the process of accessing the original video stream through the RTSP protocol is as follows: configure the accessible RTSP stream address of the video stream acquisition device, use the video stream acquisition device as the RTSP server, initiate an OPTION request to the video stream acquisition device, and obtain the media capability set of the video stream acquisition device; initiate a DESCRIBE request to the video stream acquisition device, and obtain the audio and video channel information of the video stream acquisition device after passing the device authentication; initiate a SETUP request to the video stream acquisition device to negotiate the network channel and session parameters for transmitting the video stream; initiate a PLAY request to the video stream acquisition device to notify the video stream acquisition device to start pushing the video stream. Decapsulate the video stream pushed by the video stream acquisition device to obtain the original video stream, cache the original video stream in the cache queue, and the timestamp of each video frame in the original video stream entering the cache queue is the first timestamp.

[0033] S120. Detect the original video stream based on at least one anomaly detection model of a business type to obtain an anomaly detection result of the business type; the anomaly detection result includes a second timestamp when a video frame corresponding to the anomaly detection result enters the cache queue.

[0034] Among them, the anomaly detection model is used to detect anomalies in the video stream of the on-site camera, such as: detection of not wearing a helmet, detection of personnel intrusion, detection of suspicious personnel wandering, detection of flame anomalies, detection of belt deviation, detection of gravel blockage, etc. It can be understood that different business types correspond to different anomaly detection models. Anomaly detection models include but are not limited to YOLO (You Only Look Once) series models, RCNN (Regions with Convolutional Neural Networks) series models, DETR (Detection Transformer) series models, etc., which are not limited here.

[0035] In an embodiment of the present invention, for different business types, anomaly detection is performed based on an anomaly detection model corresponding to the business type to obtain an anomaly detection result for the business type. It should be noted that not every video frame in the original video stream has an anomaly, so the anomaly detection result includes a second timestamp when the video frame corresponding to the anomaly detection result enters the cache queue. It can be understood that since the video frame corresponds to the timestamp one-to-one, the role of the second timestamp here is mainly to identify the video frame corresponding to the anomaly detection result.

[0036] On the basis of the above embodiment, optionally, the anomaly detection model based on at least one business type detects the original video stream to obtain an anomaly detection result of the business type, including: decoding the original video stream to obtain image data; inputting the image data into an anomaly detection model of at least one business type to obtain abnormal content and the abnormal location information of the business type; and determining the anomaly detection result based on the abnormal content, the abnormal location information and the second timestamp.

[0037] In an embodiment of the present invention, the original video stream is decoded to obtain image data, and the image data is format converted and preprocessed to obtain an RGB image; the RGB image is input into an anomaly detection model of at least one business type to perform anomaly detection to obtain abnormal content and abnormal location information. The abnormal content and abnormal location information are packaged and processed in combination with the first timestamp to generate an abnormal detection result.

[0038] S130: When the first timestamp and the second timestamp meet a video stream fusion condition, construct a supplementary enhancement information frame based on the anomaly detection result, and fuse the supplementary enhancement information frame with the original video stream to obtain a target video stream.

[0039] The video stream fusion condition refers to the condition for fusing the supplementary enhancement information frame with the original video stream. In an embodiment of the present invention, whether the video stream fusion condition is met is determined based on the first timestamp and the second timestamp. If the video stream fusion condition is met, a supplementary enhancement information frame is constructed based on the anomaly detection result, and the supplementary enhancement information frame and the original video stream are fused to obtain a target video stream. If the video stream fusion condition is not met, the anomaly detection result is discarded. The supplementary enhancement information frame is used to add additional information to the video stream, such as a SEI frame (Supplemental Enhancement Information). Specifically, Figure 2 As shown, the data structure of the supplementary enhancement information frame is as follows:

[0040] Set the start code of the SEI frame to be fixed to: 0x00, 0x00, 0x00, 0x01.

[0041] Set the frame type of the SEI frame to 0x06. Identify the frame as a SEI frame.

[0042] Set the payload type of the SEI frame to 0x05, which indicates that the payload content of the frame is user-defined data.

[0043] The steps to set the payload length of the SEI frame are: a) calculate the total payload data length and save it as the variable Len; b) when the value of the variable Len is less than 255, the length byte is directly set to the actual Len value, and the length assignment is completed; c) when the value of the variable Len is greater than 255, 0xFF is written to the length byte position, and the byte position index is moved one position backward; d) subtract 255 from the value of the variable Len; e) repeat steps b, c, and d.

[0044] The UUID of the SEI frame is set to the 16-bit ASCII-encoded anomaly detection identifier corresponding to the anomaly detection module, which is used to distinguish when performing SEI parsing at the application layer (ie, the front-end player).

[0045] Set the payload content of the SEI frame to a JSON format string, including the timestamp, exception content information, and exception location information (x:y:w:h) corresponding to the information.

[0046] Set the end code of the SEI frame to be fixed to: 0x80.

[0047] On the basis of the above embodiment, optionally, when the first timestamp and the second timestamp meet the video stream fusion condition, a supplementary enhancement information frame is constructed based on the anomaly detection result, and the supplementary enhancement information frame is merged with the original video stream to obtain a target video stream, including: comparing the first timestamp and the second timestamp of each video frame in the cache queue, if there is a video frame with a first timestamp identical to the second timestamp in the cache queue, then constructing a supplementary enhancement information frame based on the anomaly detection result, and inserting the supplementary enhancement information frame after the video frame corresponding to the first timestamp in the cache queue.

[0048] In an embodiment of the present invention, the first timestamp of each video frame in the cache queue of the video stream is compared with the second timestamp respectively. If there is a video frame with the first timestamp identical to the second timestamp in the cache queue, it indicates that the video frame corresponding to the abnormal detection result is still in the cache queue. A supplementary enhancement information frame is constructed based on the abnormal detection result, and the supplementary enhancement information frame is inserted into the cache queue after the video frame corresponding to the first timestamp identical to the second timestamp, so that the content of the supplementary enhancement information frame is played at the same time as the video frame before the front end plays.

[0049] On the basis of the above embodiment, optionally, when the first timestamp and the second timestamp meet the video stream fusion condition, a supplementary enhancement information frame is constructed based on the anomaly detection result, and the supplementary enhancement information frame is merged with the original video stream to obtain a target video stream, including: if there is no video frame with the same first timestamp as the second timestamp in the cache queue, then the time deviation is determined based on the first timestamp of each video frame in the cache queue and the second timestamp respectively; when the minimum time deviation in the time deviation is less than a preset minimum time deviation threshold, a supplementary enhancement information frame is constructed based on the anomaly detection result, and the supplementary enhancement information frame is inserted into the end of the cache queue.

[0050] Among them, the preset minimum time deviation threshold is used to determine whether the anomaly detection result is timely. It is understandable that it takes a certain amount of time to perform anomaly detection on the video stream based on the anomaly detection model, so the second timestamp is usually smaller than the timestamp of the latest video frame in the queue. In an embodiment of the present invention, if there is no video frame with the same first timestamp as the second timestamp in the cache queue, it indicates that the video frame corresponding to the second timestamp has been dequeued, and the time deviation is determined based on the first timestamp of each video frame in the cache queue and the second timestamp, and the minimum time deviation is determined in the time deviation; when the minimum time deviation in the time deviation is less than the preset minimum time deviation threshold, it indicates that the anomaly detection result is timely, and a supplementary enhancement information frame is constructed based on the anomaly detection result, and the supplementary enhancement information frame is inserted into the end of the cache queue. Among them, the minimum time deviation refers to the time deviation with the smallest time deviation between the first timestamp of each video frame in the cache queue and the time deviation of the second timestamp.

[0051] Based on the above embodiment, optionally, the method also includes: when the target time deviation is greater than a preset target time deviation threshold, inserting a preset number of supplementary enhancement information frames between the head of the cache queue and the supplementary enhancement information frame; the target time deviation is the time deviation between the first timestamp of the head video frame of the cache queue and the second timestamp.

[0052] In the embodiment of the present invention, when the minimum time deviation in the time deviation is less than the preset minimum time deviation threshold, the target time deviation is further compared with the preset target time deviation threshold, and if the target time deviation is greater than the preset target time deviation threshold, a preset number of supplementary enhancement information frames are inserted between the head of the cache queue and the supplementary enhancement information frame to ensure that there are enough supplementary enhancement information frames in the cache queue. The preset number is set by those skilled in the art according to needs and is not limited here.

[0053] S140: Send the target video stream to a front-end player for playback.

[0054] Based on the above embodiment, optionally, sending the target video stream to the front-end player for playback includes: in response to a stream pull request from the front-end player, sending the target video stream to the front-end player, so that the front-end player parses the target video stream, extracts anomaly detection results of the current business type, and renders the anomaly detection results into the current video screen.

[0055] In an embodiment of the present invention, an HTTP server is created. When a stream pulling request from a front-end player is received, the target video stream is sent to the front-end player in response to the stream pulling request from the front-end player. The front-end player parses the video file to obtain a supplementary enhancement information frame; based on the anomaly detection identifier in the supplementary enhancement information frame, it is determined whether the anomaly detection result in the supplementary enhancement information frame is an anomaly detection result of the current business type; if so, the supplementary enhancement information frame is parsed to obtain the anomaly detection result of the current business type, and the anomaly detection result is rendered into the current video screen. Exemplarily, the abnormal content can be added to the upper left corner of the current video screen through Canvas, and the abnormal position information can be drawn to the specified position of the current video screen through Canvas.

[0056] Based on the above embodiment, optionally, before sending the target video stream to the front-end player, the method further includes: constructing a file header of an encapsulation file; extracting a video stream identifier in the stream pull request, determining a target cache queue based on the video stream identifier, periodically reading video frames in the target cache queue, and constructing a file body of the encapsulation file based on the video frames; wherein, when reading the supplementary enhancement information frame, the counting interval judgment of reading the video frame is not involved; and determining the encapsulation file based on the file header and the file body.

[0057] Among them, the encapsulated file refers to the encapsulated target video stream. Specifically, the encapsulated file includes a file header and a file body. For example, taking the FLV (Flash Video) file as an example, the FLV header contains 9 bytes, and the first three bytes are set to fixed "F", "L" and "V" (0x46, 0x4C, 0x56). The fourth byte is set to a fixed 0x01, the 8th bit of the fifth byte is set to 1, and the other bits are set to 0, indicating that there is video data. The number of bytes between the last four bytes of the file header and the file body is set to 0x00, 0x00, 0x00, 0x09. The file body consists of video frames and tag information, and the tag information includes data frame type, frame length, timestamp and other information.

[0058] In the embodiment of the present invention, firstly, a file header of the encapsulation file is constructed; further, a video stream identifier in the pull stream request is extracted, a target cache queue is determined based on the video stream identifier, video frames are periodically read in the target cache queue, and a file body of the encapsulation file is constructed based on the video frames; further, an encapsulation file is determined based on the file header and the file body to realize encapsulation of the target video stream. It should be noted that when reading the supplementary enhancement information frame, the counting interval judgment of reading the video frame is not involved.

[0059] The technical solution of this embodiment is to access the original video stream, which includes the first timestamp of each video frame in the original video stream entering the cache queue; detect the original video stream based on at least one business type anomaly detection model to obtain the business type anomaly detection result; the anomaly detection result includes the second timestamp of the video frame corresponding to the anomaly detection result entering the cache queue; when the first timestamp and the second timestamp meet the video stream fusion conditions, construct a supplementary enhancement information frame based on the anomaly detection result, fuse the supplementary enhancement information frame with the original video stream, and obtain the target video stream; send the target video stream to the front-end player for playback. The supplementary enhancement information frame corresponding to the anomaly detection result can be fused with the original video stream to achieve real-time playback of the anomaly detection result.

[0060] Embodiment 2

[0061] Figure 3 Schematic diagram of a real-time playback system for abnormal detection results based on video streams provided by Embodiment 2 of the present invention. Figure 3 As shown, the system includes: at least one video stream acquisition device 310, a video stream anomaly detection system 320 and a front-end player 330; the video stream anomaly detection system includes a video stream access module 321, at least one video stream anomaly detection module 322, a video stream fusion module 323 and a video stream publishing module 324;

[0062] A video stream access module 321, used for accessing the original video stream collected by the video stream collection device, wherein the original video stream includes a first timestamp of each video frame in the original video stream entering a cache queue;

[0063] The video stream anomaly detection module 322 is used to detect the original video stream for any service type based on the anomaly detection model of the service type to obtain an anomaly detection result of the service type; the anomaly detection result includes a second timestamp of the video frame corresponding to the anomaly detection result entering the cache queue;

[0064] A video stream fusion module 323, configured to construct a supplementary enhancement information frame based on the anomaly detection result when the first timestamp and the second timestamp meet the video stream fusion condition, and fuse the supplementary enhancement information frame with the original video stream to obtain a target video stream;

[0065] The video stream publishing module 324 is used to send the target video stream to the front-end player for playing.

[0066] The technical solution of this embodiment is to access the original video stream, which includes the first timestamp of each video frame in the original video stream entering the cache queue; detect the original video stream based on at least one business type anomaly detection model to obtain the business type anomaly detection result; the anomaly detection result includes the second timestamp of the video frame corresponding to the anomaly detection result entering the cache queue; when the first timestamp and the second timestamp meet the video stream fusion conditions, construct a supplementary enhancement information frame based on the anomaly detection result, fuse the supplementary enhancement information frame with the original video stream, and obtain the target video stream; send the target video stream to the front-end player for playback. The supplementary enhancement information frame corresponding to the anomaly detection result can be fused with the original video stream to achieve real-time playback of the anomaly detection result.

[0067] Based on the above embodiment, optionally, the video stream anomaly detection module 322 is used to: decode the original video stream to obtain image data;

[0068] Inputting the image data into an anomaly detection model of at least one business type to obtain anomaly content and anomaly location information of the business type;

[0069] The abnormality detection result is determined based on the abnormal content, the abnormality location information and the second timestamp.

[0070] Based on the above embodiment, optionally, the video stream fusion module 323 is used to compare the first timestamp and the second timestamp of each video frame in the cache queue. If there is a video frame with the same first timestamp as the second timestamp in the cache queue, a supplementary enhancement information frame is constructed based on the anomaly detection result, and the supplementary enhancement information frame is inserted after the video frame corresponding to the first timestamp in the cache queue.

[0071] On the basis of the above embodiment, optionally, the video stream fusion module 323 is also used to determine the time deviation based on the first timestamp of each video frame in the cache queue and the second timestamp if there is no video frame with the same first timestamp as the second timestamp in the cache queue; when the minimum time deviation in the time deviation is less than a preset minimum time deviation threshold, construct a supplementary enhancement information frame based on the anomaly detection result, and insert the supplementary enhancement information frame into the end of the cache queue.

[0072] Based on the above embodiment, optionally, the video stream fusion module 323 is also used to insert a preset number of supplementary enhancement information frames between the head of the cache queue and the supplementary enhancement information frame when the target time deviation is greater than a preset target time deviation threshold; the target time deviation is the time deviation between the first timestamp of the head video frame of the cache queue and the second timestamp.

[0073] Based on the above embodiment, optionally, the video stream publishing module 324 is used to respond to the stream pulling request of the front-end player and send the target video stream to the front-end player, so that the front-end player parses the target video stream, extracts the anomaly detection result of the current business type, and renders the anomaly detection result into the current video screen.

[0074] Based on the above embodiment, optionally, the video stream publishing module 324 is further used for:

[0075] Construct the file header of the encapsulated file;

[0076] Extract the video stream identifier in the pull stream request, determine the target cache queue based on the video stream identifier, periodically read the video frames in the target cache queue, and construct the file body of the encapsulated file based on the video frames; wherein, when reading the supplementary enhancement information frame, the counting interval judgment of reading the video frame is not involved;

[0077] An encapsulated file is determined based on the file header and the file body.

[0078] The real-time playback system of anomaly detection results based on video stream provided in the embodiment of the present invention can execute the real-time playback method of anomaly detection results based on video stream provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the execution method.

[0079] Embodiment 3

[0080] Figure 41 is a schematic diagram of the structure of an electronic device provided in Embodiment 3 of the present invention. The electronic device 10 is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices (such as helmets, glasses, watches, etc.) and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present invention described and / or required herein.

[0081] like Figure 4 As shown, the electronic device 10 includes at least one processor 11, and a memory connected to the at least one processor 11, such as a read-only memory (ROM) 12, a random access memory (RAM) 13, etc., wherein the memory stores a computer program that can be executed by at least one processor, and the processor 11 can perform various appropriate actions and processes according to the computer program stored in the read-only memory (ROM) 12 or the computer program loaded from the storage unit 18 to the random access memory (RAM) 13. In the RAM 13, various programs and data required for the operation of the electronic device 10 can also be stored. The processor 11, the ROM 12, and the RAM 13 are connected to each other through a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0082] A number of components in the electronic device 10 are connected to the I / O interface 15, including: an input unit 16, such as a keyboard, a mouse, etc.; an output unit 17, such as various types of displays, speakers, etc.; a storage unit 18, such as a disk, an optical disk, etc.; and a communication unit 19, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 19 allows the electronic device 10 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0083] The processor 11 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The processor 11 executes the various methods and processes described above, such as a real-time playback method of abnormal detection results based on a video stream.

[0084] In some embodiments, the real-time playback method of the abnormal detection result based on the video stream can be implemented as a computer program, which is tangibly contained in a computer-readable storage medium, such as a storage unit 18. In some embodiments, part or all of the computer program can be loaded and / or installed on the electronic device 10 via the ROM 12 and / or the communication unit 19. When the computer program is loaded into the RAM 13 and executed by the processor 11, one or more steps of the real-time playback method of the abnormal detection result based on the video stream described above can be performed. Alternatively, in other embodiments, the processor 11 can be configured to perform the real-time playback method of the abnormal detection result based on the video stream by any other appropriate means (for example, by means of firmware).

[0085] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0086] The computer program for implementing the real-time playback method of the anomaly detection result based on video stream of the present invention can be written in any combination of one or more programming languages. These computer programs can be provided to a processor of a general-purpose computer, a special-purpose computer or other programmable data processing device, so that when the computer program is executed by the processor, the functions / operations specified in the flow chart and / or block diagram are implemented. The computer program can be executed entirely on the machine, partially on the machine, partially on the machine as an independent software package and partially on a remote machine, or completely on a remote machine or server.

[0087] Embodiment 4

[0088] Embodiment 4 of the present invention further provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to execute a method for real-time playback of abnormality detection results based on a video stream, the method comprising:

[0089] Accessing an original video stream, the original video stream including a first timestamp of each video frame in the original video stream entering a cache queue;

[0090] Detecting the original video stream based on at least one anomaly detection model of the business type to obtain an anomaly detection result of the business type; the anomaly detection result includes a second timestamp of a video frame corresponding to the anomaly detection result entering a cache queue;

[0091] When the first timestamp and the second timestamp meet the video stream fusion condition, construct a supplementary enhancement information frame based on the anomaly detection result, and fuse the supplementary enhancement information frame with the original video stream to obtain a target video stream;

[0092] Send the target video stream to the front-end player for playback.

[0093] In the context of the present invention, a computer-readable storage medium may be a tangible medium that may contain or store a computer program for use by or in combination with an instruction execution system, device or equipment. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices or equipment, or any suitable combination of the foregoing. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0094] To provide interaction with a user, the systems and techniques described herein may be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and a pointing device (e.g., a mouse or trackball) through which the user can provide input to the electronic device. Other types of devices may also be used to provide interaction with the user; for example, the feedback provided to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form (including acoustic input, voice input, or tactile input).

[0095] The systems and techniques described herein may be implemented in a computing system that includes backend components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes frontend components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such backend components, middleware components, or frontend components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), a blockchain network, and the Internet.

[0096] A computing system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The client and server relationship is generated by computer programs running on the corresponding computers and having a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and VPS services.

[0097] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps described in the present invention can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solution of the present invention can be achieved, and this document does not limit this.

[0098] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A method for real-time playback of abnormal detection results based on video stream, characterized in that: include: Accessing an original video stream, wherein the original video stream includes a first timestamp of each video frame in the original video stream entering a cache queue; Detecting the original video stream based on at least one anomaly detection model of a business type, obtaining an anomaly detection result of the business type; the anomaly detection result includes a second timestamp of a video frame corresponding to the anomaly detection result entering a cache queue; When the first timestamp and the second timestamp meet the video stream fusion condition, construct a supplementary enhancement information frame based on the anomaly detection result, and fuse the supplementary enhancement information frame with the original video stream to obtain a target video stream; The target video stream is sent to a front-end player for playback.

2. The method according to claim 1, characterized in that The detecting the original video stream based on the anomaly detection model of at least one service type to obtain an anomaly detection result of the service type includes: Decoding the original video stream to obtain image data; Inputting the image data into an anomaly detection model of at least one business type to obtain anomaly content and anomaly location information of the business type; The abnormality detection result is determined based on the abnormal content, the abnormality location information and the second timestamp.

3. The method according to claim 2, characterized in that The method further comprises: constructing a supplementary enhancement information frame based on the abnormality detection result when the first timestamp and the second timestamp meet the video stream fusion condition, and fusing the supplementary enhancement information frame with the original video stream to obtain a target video stream, including: The first timestamp and the second timestamp of each video frame in the cache queue are compared. If there is a video frame with the same first timestamp as the second timestamp in the cache queue, a supplementary enhancement information frame is constructed based on the anomaly detection result, and the supplementary enhancement information frame is inserted after the video frame corresponding to the first timestamp in the cache queue.

4. The method according to claim 3, characterized in that The method further comprises: constructing a supplementary enhancement information frame based on the abnormality detection result when the first timestamp and the second timestamp meet the video stream fusion condition, and fusing the supplementary enhancement information frame with the original video stream to obtain a target video stream, including: If there is no video frame with the same first timestamp as the second timestamp in the cache queue, determining a time offset based on the first timestamp of each video frame in the cache queue and the second timestamp respectively; When the minimum time deviation among the time deviations is less than a preset minimum time deviation threshold, a supplementary enhancement information frame is constructed based on the abnormality detection result, and the supplementary enhancement information frame is inserted into the tail of the cache queue.

5. The method according to claim 4, characterized in that The method further comprises: When the target time deviation is greater than a preset target time deviation threshold, a preset number of supplementary enhancement information frames are inserted between the head of the cache queue and the supplementary enhancement information frame; the target time deviation is the time deviation between the first timestamp of the head video frame of the cache queue and the second timestamp.

6. The method according to claim 1, characterized in that The step of sending the target video stream to a front-end player for playing includes: In response to the stream pulling request of the front-end player, the target video stream is sent to the front-end player, so that the front-end player parses the target video stream, extracts the anomaly detection result of the current business type, and renders the anomaly detection result into the current video screen.

7. The method according to claim 6, characterized in that Before sending the target video stream to the front-end player, the method further includes: Construct the file header of the encapsulated file; Extract the video stream identifier in the pull stream request, determine the target cache queue based on the video stream identifier, periodically read the video frames in the target cache queue, and construct the file body of the encapsulated file based on the video frames; wherein, when reading the supplementary enhancement information frame, the counting interval judgment of reading the video frame is not involved; An encapsulated file is determined based on the file header and the file body.

8. A real-time playback system for abnormal detection results based on video stream, characterized in that: include: At least one video stream acquisition device, a video stream anomaly detection system and a front-end player; the video stream anomaly detection system includes a video stream access module, at least one video stream anomaly detection module, a video stream fusion module and a video stream publishing module; The video stream access module is used to access the original video stream collected by the video stream acquisition device, and the original video stream includes the first timestamp of each video frame in the original video stream entering the cache queue; The video stream anomaly detection module is used to detect the original video stream for any service type based on the anomaly detection model of the service type to obtain an anomaly detection result of the service type; the anomaly detection result includes a second timestamp of the video frame corresponding to the anomaly detection result entering the cache queue; The video stream fusion module is used to construct a supplementary enhancement information frame based on the anomaly detection result when the first timestamp and the second timestamp meet the video stream fusion condition, and fuse the supplementary enhancement information frame with the original video stream to obtain a target video stream; The video stream publishing module is used to send the target video stream to the front-end player for playing.

9. An electronic device, characterized in that: The electronic device comprises: at least one processor; and a memory communicatively connected to the at least one processor; wherein, The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the real-time playback method of anomaly detection results based on video streams as described in any one of claims 1-7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the real-time playback method of anomaly detection results based on video streams according to any one of claims 1 to 7 when executed.

Images