Mosaic-video playback control system and method, electronic device, and storage medium

Abstract

Provided in the present application are a mosaic-video playback control system and method, an electronic device, and a storage medium. The system comprises a device side (10), a server side (20) and a browser side (30), wherein the device side (10) is configured to: collect video data and separately perform encoding processing and sensitive-information detection on video frames in the video data, so as to obtain an encoded frame corresponding to each video frame and sensitive information in the video frame, add the sensitive information to a protocol header of the encoded frame, so as to obtain a first video file, and upload the first video file to the server side (20); the server side (20) is configured to perform packaging processing on the first video file on the basis of a video playback request of a user, so as to obtain a second video file, and transmit the second video file to the browser side (30); and the browser side (30) is configured to receive the second video file, perform mosaic processing on video frames in the second video file, so as to obtain a mosaic video, and render and play back the mosaic video. The system can reduce storage costs, improve the video concurrency, increase the response speed, and support dynamic playback without plug-ins.

Description

Mosaic video playback control system, method, electronic device and storage medium Technical Field

[0001] The present application relates to the field of video processing technology, and in particular to a mosaic video playback control system, method, electronic device, and storage medium. Background Art

[0002] With the rapid development of technologies such as artificial intelligence (AI), video perception, lidar, and millimeter-wave radar, intelligent video surveillance systems in the transportation industry are becoming increasingly intelligent. These systems not only meet the basic needs of remote access to device audio and video resources, but also display a range of AI (artificial intelligence) information analyzed by the equipment, such as smoking, phone calls, fatigued driving, faces, vehicle models, license plates, and pedestrian information. Through big data analysis, they can provide early warnings of safety hazards and improve road safety. However, the videos captured by video surveillance systems contain private information such as license plates and faces, including information about vehicles, drivers, passengers, and pedestrians. To prevent the leakage of private information, a common method currently is to mosaic the faces and license plates in the videos. This involves uploading the video file to a server, which then performs privacy detection and mosaic processing on the video file to obtain a mosaic video file corresponding to the video file. This method cannot immediately play the mosaic video during the mosaic processing process. The identification and mosaicing of the video file takes time. The longer the video, the longer the user has to wait to play it, which can affect the playback experience for ordinary users. Moreover, this method requires storing both the original video file and the mosaic video file on the server side, which consumes twice the storage space and results in higher storage costs. Technical issues

[0003] In view of this, the embodiments of the present application provide a mosaic video playback control system, method, electronic device and storage medium, aiming to achieve at least one of the following effects: reducing server storage costs, increasing server video concurrency, improving video playback response speed, and supporting dynamic playback of mosaic videos without plug-ins. Technical Solutions

[0004] A first aspect of an embodiment of the present application provides a mosaic video playback control system, which includes a device side, a server side, and a browser side, wherein: the device side is used to collect video data and perform encoding processing and sensitive information detection on video frames in the video data, obtain the encoded frame corresponding to the video frame and the sensitive information in the video frame, add the sensitive information to the protocol header of the encoded frame, obtain a first video file, and upload the first video file to the server side; the server side is used to encapsulate the first video file uploaded by the device side according to the user's video playback request, and transmit the encapsulated second video file to the browser side; the browser side is used to receive the second video file transmitted by the server side, mosaic process the video frames in the second video file, obtain a mosaic video, and render and play the mosaic video.

[0005] In one possible implementation, the browser side includes a js layer and a wasm layer, the js layer is configured with a video streaming module and a rendering and playback module, and the wasm layer is configured with a decapsulation processing module, a video frame buffer module, a decoding processing module and a mosaic processing module, wherein: the video streaming module is used to pull the second video file from the server; the decapsulation processing module is used to decapsulate the second video file to obtain multiple video encoding frames; the video frame buffer module is used to convert the multiple video encoding frames into a single video file according to the timestamps corresponding to each of the multiple video encoding frames. The encoded video frames are sequentially cached in the buffer queue; the decoding processing module is used to decode the encoded video frames in the buffer queue in sequence to obtain the original video frames corresponding to each encoded video frame; the mosaic processing module is used to mosaic the original video frames corresponding to the encoded video frames for each encoded video frame cached in the buffer queue according to the sensitive information carried by the encoded video frame or the sensitive information carried by the associated video frames in the buffer queue that match the encoded video frame to obtain a mosaic video; the rendering and playback module is used to render and play the mosaic video.

[0006] In a possible implementation, the mosaic processing module is further configured to perform image stabilization processing on the original video frame corresponding to the video encoding frame when the video encoding frame does not carry sensitive information.

[0007] In one possible implementation, the device end includes a video acquisition module, a video encoding module, a sensitive information detection module and a video frame preprocessing module, wherein: the video acquisition module is used to perform video acquisition and obtain a video stream; the video encoding module is used to encode the video frames in the video stream to obtain video encoding frames; the sensitive information detection module is used to perform sensitive information detection on the video frames in the video stream to obtain sensitive information of the video frames; the video frame preprocessing module is used to add the sensitive information of the video frame to the protocol header of the video encoding frame to generate the first video file.

[0008] In one possible implementation, the server includes a data management module and a media service module, wherein: the data management module is used to receive the first video file uploaded by the device and save the first video file to the storage space; the media service module is used to obtain the first video file from the storage space and encapsulate the first video file in FLV format according to the user's video playback request, obtain the encapsulated second video file, and transmit the second video file to the browser.

[0009] In one possible implementation, the server further includes a device management module, wherein the device management module is configured to authenticate the device and establish a signaling link with the device to confirm the online status of the device.

[0010] A second aspect of an embodiment of the present application provides a mosaic video playback method, which is applied to the browser side of the mosaic video playback control system as described in the first aspect, and includes: pulling a second video file from a server side; decapsulating the second video file to obtain a video encoding frame in the second video file; decoding each video encoding frame in the second video file to obtain a video original frame corresponding to the video encoding frame; mosaic processing the video original frame corresponding to the video encoding frame based on sensitive information carried by the video encoding frame or sensitive information carried by an associated video frame matching the video encoding frame to obtain a mosaic video; and calling a WebGL component to render and play the mosaic video.

[0011] In one possible implementation, the step of mosaicing the original video frame corresponding to the video coding frame according to the sensitive information carried by the video coding frame or the sensitive information carried by the associated video frame matching the video coding frame to obtain a mosaic video includes: judging, for each video coding frame in the second video file, whether the video coding frame carries sensitive information according to the timestamp order of the video coding frame; if the video coding frame carries sensitive information, determining the location of the area requiring mosaic processing according to the sensitive information carried by the video coding frame, and mosaicing the original video frame corresponding to the video coding frame according to the area location; if the video coding frame does not carry sensitive information, determining the location of the area requiring mosaic processing according to the sensitive information carried by the associated video frame matching the video coding frame, and mosaicing the original video frame corresponding to the video coding frame according to the area location.

[0012] In one possible implementation, the step of determining the location of an area requiring mosaic processing based on sensitive information carried by an associated video frame that matches the video coding frame, and mosaic processing the original video frame corresponding to the video coding frame according to the location of the area includes: calculating the time difference between the timestamp of the video coding frame and the timestamp of the associated video frame that matches the video coding frame; judging whether the time difference exceeds a preset threshold; if the time difference does not exceed the preset time threshold, determining the location of an area requiring mosaic processing based on the sensitive information carried by the associated video frame that matches the video coding frame, and mosaic processing the original video frame corresponding to the video coding frame according to the location of the area.

[0013] A third aspect of an embodiment of the present application provides a mosaic video playback method, which is applied to the server side of the mosaic video playback control system as described in the first aspect, including: receiving a user's video playback request, the video playback request carrying the user's identity information; determining the user's permissions based on the user's identity information, wherein the user's permissions include ordinary user permissions and administrator user permissions; obtaining a first video file matching the video playback request from a storage space, and if the user's permissions are administrator user permissions, directly encapsulating the first video file in FLV format; if the user permissions are ordinary user permissions, extending the SEI NALU unit for the video coding frame in the first video file and adding sensitive information of the video coding frame in the SEI NALU unit body, encapsulating the video coding frame extended with the SEI NALU unit in FLV format to obtain the encapsulated second video file; and transmitting the second video file to the browser side in response to the user's video playback request.

[0014] A fourth aspect of an embodiment of the present application provides a mosaic video playback method, which is applied to the device end of the mosaic video playback control system as described in the first aspect, including: performing video capture when the device end is in the power-on state to obtain a video stream; performing encoding processing and sensitive information detection processing on the video frames in the video stream respectively to obtain a video encoding frame corresponding to the video frame and the sensitive information in the video frame; adding the sensitive information in the video frame to the protocol header of the video encoding frame to generate a first video file.

[0015] The fifth aspect of an embodiment of the present application provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and runnable on the processor, characterized in that when the processor executes the computer program, the steps of the method described in any one of the second, third, and fourth aspects are implemented.

[0016] The sixth aspect of an embodiment of the present application provides a computer-readable storage medium, which stores a computer program, and is characterized in that when the computer program is executed by a processor, it implements the steps of the method described in any one of the second aspect, the third aspect and the fourth aspect.

[0017] A seventh aspect of an embodiment of the present application provides a computer program product, which, when executed on an electronic device, enables the electronic device to implement the steps of the method described in any one of the second, third and fourth aspects. Beneficial effects

[0018] Compared with the prior art, the embodiments of the present application have the following beneficial effects:

[0019] This application uses a device side to collect video data and encode and detect sensitive information on video frames in the video data, obtains encoded frames corresponding to the video frames and sensitive information in the video frames, adds the sensitive information in the protocol header of the encoded frames, obtains a first video file, and uploads the first video file to the server side; the server side is used to encapsulate the first video file according to the user's video playback request, obtain a second video file, and transmit the second video file to the browser side; the browser side is used to receive the second video file, mosaic process the video frames in the second video file, obtain a mosaic video, and render and play the mosaic video, which can reduce the storage cost of the server side, increase the video concurrency of the server side, improve the video playback response speed, and support dynamic playback of mosaic videos without plug-ins. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the embodiments or descriptions of the prior art. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0021] FIG1 is a schematic diagram of the basic structure of a mosaic video playback control system provided in an embodiment of the present application;

[0022] FIG2 is a schematic diagram of the structure of a browser side in a mosaic video playback control system provided by an embodiment of the present application;

[0023] FIG3 is a schematic diagram of a structure of a device end in a mosaic video playback control system provided in an embodiment of the present application;

[0024] FIG4 is a schematic diagram of the structure of a server in a mosaic video playback control system provided in an embodiment of the present application;

[0025] FIG5 is a flowchart of a method for playing mosaic videos on a browser side according to an embodiment of the present application;

[0026] FIG6 is a flowchart illustrating an implementation of mosaic processing in a mosaic video playback method applied on a browser side according to an embodiment of the present application;

[0027] FIG7 is a flowchart of another implementation of mosaic processing in a mosaic video playback method applied on a browser side provided by an embodiment of the present application;

[0028] FIG8 is a schematic diagram of a video frame sequence provided in an embodiment of the present application;

[0029] FIG9 is a flowchart of an implementation method for mosaic video playback applied on a server side according to an embodiment of the present application;

[0030] FIG10 is a flowchart of an implementation method for mosaic video playback applied on a device end according to an embodiment of the present application;

[0031] FIG11 is a basic structural block diagram of an electronic device provided in an embodiment of the present application. Modes for Carrying Out the Invention

[0032] In the following description, specific details such as specific system structures and techniques are provided for purposes of illustration rather than limitation to facilitate a thorough understanding of the embodiments of the present application. However, it will be apparent to those skilled in the art that the present application may be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to avoid obscuring the description of the present application with unnecessary detail.

[0033] It should be understood that when used in the present specification and the appended claims, the term "comprising" indicates the presence of described features, integers, steps, operations, elements and / or components, but does not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or collections thereof.

[0034] It will also be understood that the term "and / or" used in this specification and the appended claims refers to and includes any and all possible combinations of one or more of the associated listed items.

[0035] As used in this specification and the appended claims, the term "if" can be interpreted as "when" or "upon" or "in response to determining" or "in response to detecting," depending on the context. Similarly, the phrase "if it is determined" or "if [described condition or event] is detected" can be interpreted as meaning "upon determination" or "in response to determining" or "upon detection of [described condition or event]" or "in response to detecting [described condition or event]," depending on the context.

[0036] In addition, in the description of the present application specification and the appended claims, the terms "first", "second", "third", etc. are only used to distinguish the descriptions and cannot be understood as indicating or implying relative importance.

[0037] References to "one embodiment" or "some embodiments" in this specification mean that one or more embodiments of the present application include a particular feature, structure, or characteristic described in conjunction with that embodiment. Thus, phrases such as "in one embodiment," "in some embodiments," "in other embodiments," and "in other embodiments" appearing in various places in this specification do not necessarily refer to the same embodiment, but rather mean "one or more but not all embodiments," unless otherwise specifically emphasized. The terms "including," "comprising," "having," and variations thereof all mean "including but not limited to," unless otherwise specifically emphasized. "Multiple" means "two or more."

[0038] In order to make the purpose, technical solutions and advantages of this application more clearly understood, the present application is further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain this application and are not intended to limit this application.

[0039] In some embodiments of the present application, please refer to Figure 1, which is a basic structural diagram of a mosaic video playback control system provided in an embodiment of the present application. As shown in Figure 1, the mosaic video playback control system mainly includes a device end 10, a server end 20 and a browser end 30. Through the interaction of the three ends, the video stream can be mosaicked and rendered while being played, thereby realizing dynamic mosaic video playback.

[0040] In this embodiment, the device 10 is used to collect video data and perform encoding and sensitive information detection on the video frames in the video data, obtain the encoded frames corresponding to the video frames and the sensitive information in the video frames, add the sensitive information to the protocol header of the encoded frames, obtain a first video file, and upload the first video file to the server. In one specific embodiment, encoding and sensitive information detection can be performed on each video frame. For each video frame, the sensitive information detected from the video frame is added to the protocol header of the encoded frame corresponding to the video frame. In another specific embodiment, sensitive information detection can be performed on the video frames in the video data according to a frame skipping strategy. Based on the time when the sensitive information is obtained, the sensitive information currently detected is added to the protocol header of the encoded frame currently encoded. In this embodiment, sensitive information such as faces and license plate numbers in the video frames can be obtained through sensitive information detection. The sensitive information includes the location information of the faces and license plate numbers in the video frames, and the location information includes information such as the horizontal coordinate, vertical coordinate, width, and height. By adding the sensitive information to the protocol header of the encoded frame, the additional storage space occupied is very small, which can save storage costs.

[0041] In this embodiment, the server 20 is used to encapsulate the first video file uploaded by the device 10 according to the user's video playback request, obtain the encapsulated second video file, and transmit the second video file to the browser 30. Specifically, after the server 20 obtains the first video file uploaded by the device 10, it can directly store the first video file in the storage space of the server 20. When the user needs to play the video file uploaded by the device, the browser 30 can request to play it, and the browser 30 will send the user's video playback request to the server 20. After receiving the user's video playback request, the server 20 can obtain the first video file that matches the video playback request from the storage space according to the video playback request, thereby encapsulating the video frames in the first video file in FLV format to obtain the encapsulated second video file, and transmit the second video file to the browser 30. It can be understood that FLV is a streaming video format developed with the launch of Flash MX, and the resulting file is small and loads quickly. It should be noted that the video frames in the second video file are prepended with an SEI NALU unit. This NALU contains sensitive information carried by the video frames, without compromising the original H264 / H265 video frame structure. This allows for audio and video data to be distributed using standard streaming protocols such as HTTP-FLV, HLS, and RTMP, without compromising the standards of these protocols. This ensures excellent versatility and scalability. It's understood that SEI (Supplemental Enhancement Information) provides a method for adding additional information to the video bitstream and is a feature of the H264 / H265 video compression standards. NALU (Network Abstraction Layer Unit) encapsulates data provided by the video coding layer for network transmission. HTTP-FLV (Flash Video Over HTTP) is a protocol for transmitting FLV audio and video data over HTTP. HLS (Http Live Streaming) is a streaming network transmission protocol based on HTTP. RTMP (Real Time Messaging Protocol) is an open protocol for direct audio, video, and data transmission between Flash players and servers. The server 20 only needs to provide FLV format encapsulation and video data forwarding functions in the mosaic video playback control system, which occupies less CPU (Central Processing Unit) and GPU (Graphics Processing Unit) resources and can increase the video concurrency of the server 20.Since the device end 10 has identified the sensitive information and attached it to the protocol header of the video frame, there is no need for the server end 20 to perform information identification. It only needs to be encapsulated in FLV format and transmitted to the browser end 30, which can improve the response speed of video playback.

[0042] In this embodiment, the browser 30 is configured to receive a second video file transmitted from the server 20, perform mosaic processing on the video frames in the second video file to obtain a mosaic video, and then render and play the mosaic video. Specifically, the browser 30, acting as a video playback client, can implement human-computer interaction with the user. The user can initiate a video playback request by performing relevant operations in the interactive interface of the browser 30. The browser 30 then sends the video playback request to the server 20, thereby retrieving the desired video file. The desired video file is the second video file encapsulated by the server 20. After receiving the second video file, the browser 30 decapsulates the second video file to obtain the video frames in the second video file. The obtained video frames are represented as encoded video frames. The encoded video frames are decoded to obtain the original H264 / H265 video frames. By parsing the SEI NALU unit that extends the front of the encoded video frame, sensitive information recorded in the SEI NALU unit can be obtained. Furthermore, the original video frame is mosaic processed based on the sensitive information to obtain the mosaic video. Furthermore, by calling the WebGL component of the browser side 30, the mosaic video can be rendered and played. It can be understood that WebGL (Web Graphics Library) is a 3D drawing protocol that can provide hardware 3D accelerated rendering. Based on the above embodiment, the mosaic video playback control system moves the face and license plate number detection in the video from the server side 20 to the device side 10, and the video mosaic processing from the server side 20 to the browser side 30. The edge computing and browser page front-end processing mode are used to avoid video processing on the server side 20. The server side 20 only needs to provide FLV format encapsulation and data forwarding, which occupies very few CPU and GPU resources, and the video concurrency is greatly improved.

[0043] In some embodiments of the present application, please refer to Figure 2, which is a structural diagram of the browser side in the mosaic video playback control system provided in the embodiment of the present application. As shown in Figure 2, the browser side 30 includes a js layer 21 and a wasm layer 22. Specifically, the plug-in-free playback library of the H5 page in the browser is physically a js library, and the js library is logically divided into a js layer 21 and a wasm layer 22. The js layer 21 is written in JavaScript, and the wasm layer 22 is compiled using WebAssembly technology. In this embodiment, the js layer 21 is configured with a video streaming module 211 and a rendering and playback module 212. Specifically, the video streaming module 211 is used to pull the second video file from the server 20. The rendering and playback module 212 is used to render and play the mosaic video. The wasm layer 22 is configured with a decapsulation processing module 221, a video frame buffer module 222, a decoding processing module 223, and a mosaic processing module 224. Specifically, the decapsulation processing module 221 is configured to decapsulate the second video file to obtain multiple video encoding frames. It should be noted that all or some of the multiple video encoding frames may carry sensitive information. The video frame buffering module 222 is configured to sequentially cache the multiple video encoding frames in a buffer queue according to their corresponding timestamps. The decoding processing module 223 is configured to sequentially decode the video encoding frames in the buffer queue to obtain the original video frame corresponding to each video encoding frame. The mosaic processing module 224 is configured to mosaic the original video frame corresponding to each video encoding frame cached in the buffer queue based on the sensitive information carried by the video encoding frame or the sensitive information carried by the associated video frames in the buffer queue that match the video encoding frame, thereby obtaining a mosaic video. In this embodiment, the browser 30 compiles the C / C++ program, including video decapsulation, decoding, mosaic processing, and video frame buffering, into the wasm layer 22 using WebAssembly technology, which provides high security and maximizes user privacy. By calling the js layer 21 to render and play the mosaic video, performance and security requirements can be taken into account at the same time. The wasm layer 22 can be published together with the web page program, without the need to install other plug-ins separately on the browser side 30, and supports plug-in-free dynamic mosaic video playback on the browser side 30.

[0044] In some embodiments of the present application, please refer to Figure 3, which is a structural diagram of the device end in the mosaic video playback control system provided in the embodiments of the present application. As shown in Figure 3, the device end 10 is configured with a video acquisition module 31, a video encoding module 32, a sensitive information detection module 33, and a video frame preprocessing module 34. In this embodiment, the video acquisition module 31 is used to perform video acquisition and obtain a video stream. The video encoding module 32 is used to encode the video frames in the video stream to obtain video encoding frames. The sensitive information detection module 33 is used to perform sensitive information detection on the video frames in the video stream to obtain sensitive information of the video frames. The video frame preprocessing module 34 is used to add the sensitive information of the video frame to the protocol header of the video encoding frame to generate a first video file. Specifically, the device end 10, as the producer of the video file, can control the camera to perform video acquisition through the video acquisition module to obtain a video stream. In this embodiment, the video stream is represented as a video frame sequence, and the video frames in the video frame sequence are sorted according to timestamps. After the device 10 receives the video stream, the video encoding module 32 sequentially performs H264 / H265 video encoding on each video frame in the video stream according to the timestamp, obtaining the video encoded frame corresponding to each video frame. Simultaneously, the sensitive information detection module 33 performs sensitive information detection on some or all video frames in the video stream, obtaining sensitive information from some or all video frames. In this embodiment, the video encoding module 32 and the sensitive information detection module 33 are executed simultaneously using two independent threads. In this embodiment, the video encoding module 32 sequentially encodes each video frame in the video stream. To conserve CPU and GPU resources on the device 10, the sensitive information detection module 33 performs sensitive information detection on the video frames in the video stream according to a pre-defined frame skipping recognition strategy, i.e., only obtaining sensitive information from a portion of the video frames. For example, the video encoding module 32 encodes the video frames at a frame rate of 25 frames per second, while the sensitive information detection module 33 recognizes 10 to 16 frames per second. In this embodiment, when the sensitive information detection module 33 detects sensitive information, in order to avoid affecting the real-time performance of the video, the video frame preprocessing module 34 will add the sensitive information of the video frame currently detected by the sensitive information detection module and the timestamp of the detected video frame to the protocol header of the encoded video frame currently encoded by the video encoding module.

[0045] In some embodiments of the present application, since the video encoding module 32 and the sensitive information detection module 33 of the device end 10 are executed simultaneously using two independent threads, and the sensitive information detection module 33 implements frame skipping recognition, the sensitive information carried in the protocol header of the current video frame in the first video file belongs to the historical video frame. That is, the first video file may contain some video frames that carry sensitive information, while some video frames do not carry sensitive information. Based on this, in this embodiment, the mosaic processing module 224 in the browser end 30 is also used to perform image stabilization processing on the original video frame corresponding to the video encoding frame when the video encoding frame does not carry sensitive information. Specifically, the process logic of the picture anti-shake processing is as follows: if the first frame of video does not carry sensitive information, the time difference between the timestamp of the first frame of video and the timestamp of the associated video frame that matches the first frame of video is calculated, and it is determined whether the time difference exceeds the preset threshold. If the time difference does not exceed the preset threshold, the first frame of video is mosaiced using the area represented by the sensitive information carried by the associated video frame. If the time difference exceeds the preset threshold, the first frame of video is not mosaiced. If the non-first frame of video does not carry sensitive information, the time difference between the timestamp of the non-first frame of video and the timestamp of the associated video frame that matches the first frame of video is calculated, and it is determined whether the time difference exceeds the preset threshold. If the time difference does not exceed the preset threshold, the non-first frame of video is mosaiced using the area represented by the sensitive information carried by the associated video frame. If the time difference exceeds the preset threshold, the non-first frame of video is not mosaiced. It can be understood that in this embodiment, whether it is the first frame video frame or the non-first frame video frame, when the video frame carries sensitive information, the video frame is directly mosaicked according to the area position represented by the sensitive information.

[0046] In some embodiments of the present application, please refer to Figure 4, which is a structural diagram of the server in the mosaic video playback control system provided in the embodiments of the present application. As shown in Figure 4, the server 20 is configured with a data management module 41 and a media service module 42. In this embodiment, the data management module 41 is used to receive the first video file uploaded by the device and save the first video file to the storage space. The media service module 42 is used to obtain the first video file from the database and encapsulate the first video file in FLV format according to the user's video playback request, and transmit the encapsulated second video file to the browser. Specifically, when encapsulating the first video file in FLV format, the media service module 42 can confirm the user's authority based on the user's video playback request. If the user is confirmed to have ordinary user authority, an SEI NALU unit is extended in front of the video frame, and sensitive information is added to the NALU unit body before encapsulation in FLV format; if the user is confirmed to have administrator user authority, the video frame is directly encapsulated in FLV format.

[0047] In some embodiments of the present application, the server 20 is further configured with a device management module, which is used to authenticate the device and establish a signaling link with the device to confirm the online status of the device. In this embodiment, the device 10 will automatically connect to the server 20 after startup and request to establish a signaling link for communication between the device 10 and the server 20. The server 20 authenticates the device 10 requesting a connection through the device management module. When the authentication is successful, a signaling link for communication between the device 10 and the server 20 will be established, and the device 10 will be set to an online state. At this time, the device 10 can actively report the generated first video file to the server 20.

[0048] In some embodiments of the present application, please refer to Figure 5, which is a flowchart of a mosaic video playback method applied on a browser side provided by an embodiment of the present application. As shown in Figure 5, it can specifically include steps S51 to S55.

[0049] S51: Pull the second video file from the server;

[0050] S52: Decapsulate the second video file to obtain video encoding frames in the second video file;

[0051] S53: For each video encoding frame in the second video file, decode the video encoding frame to obtain an original video frame corresponding to the video encoding frame;

[0052] S54: mosaicing the original video frame corresponding to the video encoding frame according to the sensitive information carried by the video encoding frame or the sensitive information carried by the associated video frame matching the video encoding frame to obtain a mosaic video;

[0053] S55: Calling the WebGL component to render and play the mosaic video.

[0054] In this embodiment, on the browser side, a user can initiate a video playback request by inputting an operation on the browser's interactive page, thereby requesting the playback of the desired video file. After the user initiates the video playback request, the browser side sends the video playback request to the server side, which then retrieves a second video file matching the video playback request from the server side via a streaming media transmission protocol such as HTTP-FLV, HLS, or RTMP. Because the second video file is a video stream encoded on the device side and encapsulated in FLV format on the server side, after receiving the second video file from the server side, the browser side must first decapsulate the second video file to obtain all video encoding frames in the second video file. It then decodes each video encoding frame in the second video file to obtain the original video frame corresponding to each video encoding frame. Furthermore, by performing SEI parsing on each video encoding frame in the second video file, it is determined whether each video encoding frame carries sensitive information. For video encoding frames that carry sensitive information, the sensitive information in the video encoding frame is retrieved. It should be noted that the sensitive information in the video encoding frame is represented as region location information, including information such as the region's horizontal coordinate, vertical coordinate, width, and height. For example, the horizontal coordinate, vertical coordinate, width and height of the area where the face is located in the video frame, or the horizontal coordinate, vertical coordinate, width and height of the area where the license plate number is located in the video. After the browser obtains the original video frames corresponding to each video encoding frame in the second video file and the sensitive information carried, it can mosaic the original video frames corresponding to each video encoding frame in sequence according to the timestamp order of each video encoding frame, according to the sensitive information carried by the video encoding frame or the sensitive information carried by the associated video frame that matches the video encoding frame, to obtain a mosaic video. Then, the WebGL component in the browser js library is called to render and play the obtained mosaic video, so that the browser can realize mosaic processing and rendering and playing of the video stream at the same time.

[0055] It can be understood that in this embodiment, if all the video encoding frames in the second video file do not carry sensitive information, there is no need to mosaic the video encoding frames. The video encoding frames can be directly decoded according to the timestamp order of each video encoding frame and then rendered and played.

[0056] In some embodiments of the present application, please refer to Figure 6, which is a flowchart of an implementation of mosaic processing in a mosaic video playback method applied on a browser side provided by an embodiment of the present application. As shown in Figure 6, it can specifically include steps S61 to S63.

[0057] S61: Determine, for each video encoding frame in the second video file, according to the timestamp order of the video encoding frames, whether the video encoding frame carries sensitive information;

[0058] S62: If the video encoding frame carries sensitive information, determine the location of a region requiring mosaic processing according to the sensitive information carried by the video encoding frame, and perform mosaic processing on the original video frame corresponding to the video encoding frame according to the location of the region;

[0059] S63: If the video encoding frame does not carry sensitive information, determine the area position that needs to be mosaiced based on the sensitive information carried by the associated video frame that matches the video encoding frame, and perform mosaicing on the original video frame corresponding to the video encoding frame according to the area position.

[0060] In this embodiment, to conserve CPU and GPU resources on the device side, the device side uses a frame skipping identification strategy to detect sensitive information in the video stream. Specifically, when the browser obtains the second video file, only some of the video encoding frames will carry sensitive information. In this embodiment, the browser performs mosaic processing frame by frame according to the timestamp order of the video encoding frames. For each video encoding frame, it is first determined whether the video encoding frame carries sensitive information. Specifically, after the browser obtains the video encoding frame through decapsulation, it further uses SEI parsing to identify the extended SEI NALU units in the video encoding frame. If the main body of the SEI NALU unit records the sensitive information of the video encoding frame, it indicates that the video encoding frame carries sensitive information. If the SEI parsing process does not parse the extended SEI NALU unit, it indicates that the video encoding frame does not carry sensitive information. If the video encoding frame carries sensitive information, the location of the area in the video encoding frame that requires mosaic processing can be directly determined based on the sensitive information carried by the video encoding frame, and mosaic processing is performed on the original video frame corresponding to the video encoding frame according to the location of the area. If the video coding frame does not carry sensitive information, the area position that needs to be mosaiced in the video coding frame can be determined based on the sensitive information carried by the associated video frame that matches the video coding frame, and the original video frame corresponding to the video coding frame can be mosaiced according to the area position.

[0061] In some embodiments of the present application, please refer to Figure 7, which is another flowchart of implementing mosaic processing in the mosaic video playback method applied in the browser provided by the embodiment of the present application. As shown in Figure 7, it can specifically include steps S71 to S72.

[0062] S71: Calculate a time difference between a timestamp of the video coding frame and a timestamp of an associated video frame that matches the video coding frame;

[0063] S72: Determine whether the time difference exceeds a preset threshold. If the time difference does not exceed the preset time threshold, determine the area position that needs to be mosaiced based on the sensitive information carried by the associated video frame that matches the video encoding frame, and perform mosaic processing on the original video frame corresponding to the video encoding frame according to the area position.

[0064] In this embodiment, please refer to FIG8 , which is a schematic diagram of a video frame sequence provided in an embodiment of the present application. In this embodiment, the browser can set a video frame buffer queue for caching the video encoding frames decapsulated from the second video file in timestamp order. If the video encoding frames carry sensitive information, the sensitive information is bound to the video encoding frames and cached in the buffer queue. After receiving the video encoding frames, the browser needs to buffer for a certain time value, which is a preset threshold set by the user. If the time is lower than the preset threshold, it needs to wait for playback. In this embodiment, during the mosaic processing of the video encoding frames, if the video encoding frames do not carry sensitive information, the time difference between the timestamp of the video encoding frame and the timestamp of the associated video frame that matches the video encoding frame is calculated. By determining whether the time difference exceeds the preset threshold, if the time difference does not exceed the preset threshold, the location of the area requiring mosaic processing is determined based on the sensitive information carried by the associated video frame that matches the video encoding frame, and then mosaic processing is performed on the original video frame corresponding to the video encoding frame according to the location of the area. Exemplarily, as shown in FIG8 , on the browser side, the received video coding frames are sorted according to the timestamps. Assuming that the timestamp of the first video coding frame is pts1, the previous video coding frame carrying sensitive information is determined to be an associated video coding frame that matches the first video coding frame, and the timestamp is pts2. The timestamp of the non-first video coding frame is pts3. The preset threshold is set to 200 milliseconds. If the coded video frame currently undergoing mosaic processing is the first frame, the timestamp pts1 of the first video coding frame is subtracted from the timestamp pts2 of the previous video coding frame carrying sensitive information to obtain the time difference. When pts1-pts2≤200ms, the mosaic area of ​​the previous video coding frame carrying sensitive information is used to code the first video coding frame. When pts1-pts2>200ms, the first video coding frame is not coded. If the coded video frame currently undergoing mosaic processing is not the first frame, the timestamp pts3 of the non-first frame is subtracted from the timestamp pts2 of the previous coded video frame carrying sensitive information to obtain the time difference. If pts3 - pts2 ≤ 200ms, the mosaic area of ​​the previous coded video frame carrying sensitive information is used to mosaic the non-first frame. If pts1 - pts2 > 200ms, the non-first frame is not mosaicked.

[0065] In some embodiments of the present application, please refer to Figure 9, which is a flowchart of a mosaic video playback method applied on a server provided in an embodiment of the present application. As shown in Figure 9, it can specifically include steps S91 to S94.

[0066] S91: Receive a video playback request from a user, where the video playback request carries the user's identity information;

[0067] S92: Determine the user's rights based on the user's identity information, where the user's rights include common user rights and administrator user rights;

[0068] S93: Obtain a first video file that matches the video playback request from the storage space. If the user has administrator user rights, directly encapsulate the first video file in FLV format. If the user has ordinary user rights, expand the SEI NALU unit for the video coding frame in the first video file and add sensitive information of the video coding frame in the SEI NALU unit body. Encapsulate the video coding frame with the extended SEI NALU unit in FLV format to obtain the encapsulated second video file.

[0069] S94: In response to the user's video playback request, the second video file is transmitted to the browser.

[0070] In this embodiment, the need for mosaic processing of the played video can be determined based on the user's permissions. Specifically, a user initiates a video play request by performing relevant operations in the browser's interactive interface. The video send request carries the user's identity information and information about the video source to be played. The browser then sends the video play request to the server. The user's identity information can include information such as an account ID and identity ID. The video source information to be played includes information about the video's source device and the video's time range. After receiving the user's video play request, the server can determine the user's permissions based on the user's identity information. Specifically, user permissions include ordinary user permissions and administrator user permissions. The server stores a list of administrator users, which contains the identity information of all administrator users. In this embodiment, the list of administrator users can be traversed based on the user's identity information to determine whether the user is an administrator user. If the user is an administrator user, the user's permissions are determined to be administrator user permissions; otherwise, the user's permissions are determined to be ordinary user permissions. After determining the user's permissions, the server can retrieve the first video file that matches the video play request from the server's storage space based on the video source information to be played carried in the video send request. When encapsulating the first video file in FLV format, if the user's authority is that of an administrator, the first video file is directly encapsulated in FLV format. If the user's authority is that of a normal user, the SEI NALU unit is expanded for the video coding frame in the first video file and sensitive information of the video coding frame is added to the body of the SEI NALU unit, and the video coding frame extended with the SEI NALU unit is then encapsulated in FLV format. The encapsulated second video file can be obtained through FLV format encapsulation. After the server encapsulates the second video file, it can respond to the user's video playback request and transmit the second video file to the browser using standard streaming media transmission protocols such as HTTP-FLV, HLS, and RTMP.

[0071] In some embodiments of the present application, please refer to Figure 10, which is a flowchart of a mosaic video playback method applied on a device provided in an embodiment of the present application. As shown in Figure 10, it can specifically include steps S101 to S103.

[0072] S101: Capture video when the device is powered on to obtain a video stream.

[0073] S102: performing encoding processing and sensitive information detection processing on the video frames in the video stream respectively to obtain a video encoding frame corresponding to the video frame and sensitive information in the video frame;

[0074] S103: Add the sensitive information in the video frame to the protocol header of the video encoding frame to generate a first video file.

[0075] In this embodiment, the device side acts as the producer of the video file. After the device side is turned on, it will collect video data within the visual range through the camera in real time to obtain a video stream. At this time, the format of the video frames in the video stream is YUV format. The device side is configured with two threads, one of which is used to perform H264 / H265 encoding processing on the collected YUV format video frames frame by frame to obtain corresponding video encoding frames. The other thread is used to perform sensitive information detection processing on the collected YUV format video frames to obtain sensitive information in the video frames. In some specific implementations, a frame skipping recognition strategy can be used to perform sensitive information detection processing on the collected YUV format video frames. When it is detected that the current video frame contains sensitive information such as human faces and license plates, the detected sensitive information is added to the video frame protocol header to generate the first video file.

[0076] It can be understood that the size of the serial numbers of the steps in the above embodiments does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0077] In some embodiments of the present application, please refer to Figure 11, which is a basic structural block diagram of an electronic device provided in an embodiment of the present application. As shown in Figure 11, the electronic device 11 of this embodiment includes: a processor 111, a memory 112, and a computer program 113 stored in the memory 112 and executable on the processor 111, such as the programs of the above-mentioned mosaic video playback methods. When the processor 111 executes the computer program 113, the steps in each embodiment of the above-mentioned mosaic video playback method are implemented. Alternatively, when the processor 111 executes the computer program 113, the functions of each module in the embodiment corresponding to the above-mentioned mosaic video playback control system are implemented. Please refer to the relevant description in the embodiment for details, which will not be repeated here.

[0078] Exemplarily, the computer program 113 may be divided into one or more modules (units), which are stored in the memory 112 and executed by the processor 111 to implement the present application. The one or more modules may be a series of computer program instruction segments capable of performing specific functions, which describe the execution process of the computer program 113 in the electronic device 11. For example, the computer program 113 may be divided into the modules described above in the mosaic video playback control system, with the functions of each module being as described above.

[0079] The electronic device may include, but is not limited to, a processor 111 and a memory 112. Those skilled in the art will appreciate that FIG11 is merely an example of the electronic device 11 and does not limit the electronic device 11. The electronic device 11 may include more or fewer components than shown in the figure, or may combine certain components or different components. For example, the electronic device may further include input and output devices, network access devices, buses, and the like.

[0080] The processor 111 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor.

[0081] The memory 112 may be an internal storage unit of the electronic device 11, such as a hard drive or memory of the electronic device 11. The memory 112 may also be an external storage device of the electronic device 11, such as a plug-in hard drive, a Smart Media Card (SMC), a Secure Digital (SD) card, a flash memory card, etc. equipped on the electronic device 11. Furthermore, the memory 112 may include both an internal storage unit of the electronic device 11 and an external storage device. The memory 112 is used to store the computer program and other programs and data required by the electronic device. The memory 112 may also be used to temporarily store data that has been output or is about to be output.

[0082] It should be noted that the information interaction, execution process, etc. between the above-mentioned modules / units are based on the same concept as the method embodiment of this application. Their specific functions and technical effects can be found in the method embodiment section and will not be repeated here.

[0083] The present application also provides a computer-readable storage medium that stores a computer program. When the computer program is executed by a processor, the computer-readable storage medium can implement the steps of the above-mentioned method embodiments. In this embodiment, the computer-readable storage medium can be non-volatile or volatile.

[0084] An embodiment of the present application provides a computer program product. When the computer program product is run on a mobile terminal, the mobile terminal can implement the steps of the above-mentioned various method embodiments when executing the computer program product.

[0085] Those skilled in the art can clearly understand that, for the convenience and brevity of description, only the division of the above-mentioned functional units and modules is used as an example for illustration. In actual applications, the above-mentioned functions can be distributed and completed by different functional units and modules as needed, that is, the internal structure of the system can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiment can be integrated into one processing unit, or each unit can exist physically alone, or two or more units can be integrated into one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of this application. The specific working process of the units and modules in the above-mentioned system can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here.

[0086] If the integrated module / unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present application can implement all or part of the process steps in the above-mentioned method embodiments by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium. When executed by a processor, the computer program can implement the steps of each of the above-mentioned method embodiments. The computer program includes computer program code, which can be in source code form, object code form, executable file, or some intermediate form. The computer-readable medium can include: any entity or device capable of carrying the computer program code, recording medium, USB flash drive, mobile hard drive, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signal, telecommunication signal, and software distribution medium. It should be noted that the content of the computer-readable medium can be appropriately increased or decreased based on the requirements of legislation and patent practice in a jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable media does not include electric carrier signals and telecommunication signals.

[0087] In the above embodiments, the description of each embodiment has its own focus. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant description of other embodiments.

[0088] The above-described embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. These modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present application, and should all be included in the scope of protection of the present application.

Claims

1. A mosaic video playback control system, characterized in that: The system includes a device side, a server side and a browser side, wherein: The device end is used to collect video data and perform encoding processing and sensitive information detection on video frames in the video data, obtain an encoded frame corresponding to the video frame and sensitive information in the video frame, add the sensitive information to a protocol header of the encoded frame, obtain a first video file, and upload the first video file to the server end; The server is used to encapsulate the first video file uploaded by the device according to the video playback request of the user, and transmit the second video file obtained after encapsulation to the browser; The browser end is used to receive the second video file transmitted by the server end, perform mosaic processing on the video frames in the second video file to obtain a mosaic video, and render and play the mosaic video.

2. The mosaic video playback control system according to claim 1, characterized in that: The browser end includes a js layer and a wasm layer, wherein the js layer is configured with a video streaming module and a rendering and playing module, and the wasm layer is configured with a decapsulation processing module, a video frame buffer module, a decoding processing module and a mosaic processing module, wherein: The video streaming module is used to pull the second video file from the server; The decapsulation processing module is used to decapsulate the second video file to obtain multiple video encoding frames; The video frame buffer module is used to sequentially buffer the multiple video encoding frames into a buffer queue according to the timestamps corresponding to the multiple video encoding frames respectively; The decoding processing module is used to decode the video encoding frames in the buffer queue in order to obtain the video original frame corresponding to each video encoding frame; The mosaic processing module is used to perform mosaic processing on the original video frame corresponding to each video encoding frame cached in the buffer queue according to the sensitive information carried by the video encoding frame or the sensitive information carried by the associated video frame matching the video encoding frame in the buffer queue to obtain a mosaic video; The rendering and playing module is used to render and play the mosaic video.

3. The mosaic video playback control system according to claim 2, characterized in that: The mosaic processing module is also used to perform image stabilization processing on the original video frame corresponding to the video encoding frame when the video encoding frame does not carry sensitive information.

4. The mosaic video playback control system according to claim 1, characterized in that: The device end includes a video acquisition module, a video encoding module, a sensitive information detection module and a video frame preprocessing module, wherein: The video acquisition module is used to acquire video streams. The video encoding module is used to encode the video frames in the video stream to obtain video encoding frames; The sensitive information detection module is used to perform sensitive information detection on the video frames in the video stream to obtain sensitive information of the video frames; The video frame preprocessing module is used to add the sensitive information of the video frame to the protocol header of the video encoding frame to generate the first video file.

5. The mosaic video playback control system according to claim 1, characterized in that: The server includes a data management module and a media service module, wherein: The data management module is used to receive the first video file uploaded by the device and save the first video file to the storage space; The media service module is used to obtain the first video file from the storage space and encapsulate the first video file in FLV format according to the user's video playback request, obtain the encapsulated second video file, and transmit the second video file to the browser end.

6. The mosaic video playback control system according to claim 5, characterized in that: The server also includes a device management module, wherein: The device management module is used to authenticate the device end and establish a signaling link with the device end to confirm the online status of the device end.

7. A mosaic video playback method, characterized in that: The method is applied to the browser side of the mosaic video playback control system according to any one of claims 1 to 6, comprising: Pull the second video file from the server; Decapsulate the second video file to obtain a video encoding frame in the second video file; For each video encoding frame in the second video file, decoding the video encoding frame to obtain a video original frame corresponding to the video encoding frame; According to the sensitive information carried by the video encoding frame or the sensitive information carried by the associated video frame matching the video encoding frame, mosaic processing is performed on the original video frame corresponding to the video encoding frame to obtain a mosaic video; The WebGL component is called to render and play the mosaic video.

8. The mosaic video playing method according to claim 7, characterized in that: The step of performing mosaic processing on the original video frame corresponding to the video encoding frame according to the sensitive information carried by the video encoding frame or the sensitive information carried by the associated video frame matching the video encoding frame to obtain the mosaic video includes: According to the timestamp order of the video coding frames, for each video coding frame in the second video file, determining whether the video coding frame carries sensitive information; If the video encoding frame carries sensitive information, determine the area position that needs to be mosaic processed according to the sensitive information carried by the video encoding frame, and perform mosaic processing on the original video frame corresponding to the video encoding frame according to the area position; If the video encoding frame does not carry sensitive information, the area position requiring mosaic processing is determined according to the sensitive information carried by the associated video frame matching the video encoding frame, and the original video frame corresponding to the video encoding frame is mosaic processed according to the area position.

9. The mosaic video playing method according to claim 8, characterized in that: The step of determining the location of an area that needs to be mosaicked based on sensitive information carried by an associated video frame that matches the video coding frame, and mosaicking the original video frame corresponding to the video coding frame according to the location of the area includes: Calculating a time difference between a timestamp of the video coding frame and a timestamp of an associated video frame that matches the video coding frame; Determine whether the time difference exceeds a preset threshold. If the time difference does not exceed the preset time threshold, determine the area position that needs to be mosaic processed based on the sensitive information carried by the associated video frame that matches the video encoding frame, and perform mosaic processing on the original video frame corresponding to the video encoding frame according to the area position.

10. A mosaic video playback method, characterized in that: The method is applied to the server of the mosaic video playback control system according to any one of claims 1 to 6, comprising: Receive a video playback request from a user, where the video playback request carries the identity information of the user; Determining the rights of the user according to the identity information of the user, wherein the rights of the user include ordinary user rights and administrator user rights; A first video file matching the video playback request is obtained from the storage space. If the user has the administrator user authority, the first video file is directly encapsulated in the FLV format. If the user has the ordinary user authority, a SEI NALU unit is extended for the video coding frame in the first video file, sensitive information of the video coding frame is added to the SEI NALU unit body, and the video coding frame extended with the SEI NALU unit is encapsulated in the FLV format to obtain the encapsulated second video file. In response to the user's video playback request, the second video file is transmitted to the browser end.

11. A mosaic video playback method, characterized in that: The method is applied to a device end of the mosaic video playback control system according to any one of claims 1 to 6, comprising: Capture video when the device is powered on to obtain video stream; Performing encoding processing and sensitive information detection processing on the video frames in the video stream respectively, to obtain video encoding frames corresponding to the video frames and sensitive information in the video frames; The sensitive information in the video frame is added to a protocol header of the video encoding frame to generate a first video file.

12. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 7 to 9 or 10 or 11 are implemented.

13. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 7 to 9 or 10 or 11 are implemented.

Images