Mosaic video playback control system, method, electronic device, and storage medium

Abstract

The application provides a mosaic video playing control system, method, electronic device and storage medium. The system comprises a device end, a server end and a browser end. The device end is used for collecting video data, encoding and processing video frames in the video data and detecting sensitive information in the video frames, obtaining encoded frames corresponding to the video frames and the sensitive information in the video frames, adding the sensitive information in a protocol header of the encoded frames, obtaining a first video file, and uploading the first video file to the server end. The server end is used for performing encapsulation processing on the first video file according to a video playing request of a user, obtaining a second video file, and transmitting the second video file to the browser end. The browser end is used for receiving the second video file, performing mosaic processing on video frames in the second video file, obtaining a mosaic video, and rendering and playing the mosaic video. The system can reduce storage cost, improve video concurrency, improve response speed and support plug-in-free dynamic playing.

Description

Technical Field

[0001] This application relates to the field of video processing technology, and in particular to a mosaic video playback control system, method, electronic device and storage medium. Background Technology

[0002] With the rapid development of technologies such as artificial intelligence, video perception, LiDAR, and millimeter-wave radar, intelligent video surveillance systems in the transportation industry are becoming increasingly intelligent. They not only meet the basic needs of remotely accessing device audio and video resources but also possess the ability to display a range of AI (Artificial Intelligence) information analyzed by the devices, such as smoking, phone calls, fatigued driving, facial recognition, vehicle models, license plate numbers, and pedestrian information. Through big data analysis, they can provide early warnings of safety hazards, improving road safety. However, the videos collected by video surveillance systems contain private information about vehicles, drivers, passengers, and pedestrians, such as license plates and faces. To prevent the leakage of this privacy information, a common method is to blur the faces and license plates in the video. This involves uploading the video file to a server, where the server performs privacy detection and blurring to obtain a blurred video file. However, this method cannot immediately play the blurred video during the blurring process; the identification and blurring of the blurred areas takes time. The longer the video, the longer the waiting time for users, potentially impacting the viewing experience for ordinary users. Moreover, this method requires storing both the original video file and the mosaic video file on the server side, which consumes twice the storage space and results in higher storage costs. Summary of the Invention

[0003] In view of this, embodiments of this application provide a mosaic video playback control system, method, electronic device, and storage medium, aiming to achieve at least one of the following effects: reducing server-side storage costs, increasing server-side video concurrency, improving video playback response speed, and supporting plugin-free dynamic playback of mosaic videos.

[0004] A first aspect of this application provides a mosaic video playback control system, the system comprising a device, a server, and a browser, wherein: the device is configured to acquire video data and perform encoding processing and sensitive information detection on video frames in the video data respectively, obtain the encoded frame corresponding to the video frame and the sensitive information in the video frame, add the sensitive information to the protocol header of the encoded frame to obtain a first video file, and upload the first video file to the server; the server is configured to encapsulate the first video file uploaded by the device according to the user's video playback request, and transmit the encapsulated second video file to the browser; the browser is configured to receive the second video file transmitted by the server, perform mosaic processing on the video frames in the second video file to obtain a mosaic video, and render and play the mosaic video.

[0005] In one possible implementation, the browser includes a JS layer and a WASM layer. The JS layer is configured with a video streaming module and a rendering and playback module. The WASM layer is configured with a decapsulation processing module, a video frame buffering module, a decoding processing module, and a mosaic processing module. Specifically: the video streaming module is used to retrieve a second video file from the server; the decapsulation processing module is used to decapsulate the second video file to obtain multiple video encoded frames; and the video frame buffering module is used to buffer the multiple video encoded frames according to their respective timestamps. The video encoded frames are sequentially cached into a buffer queue. The decoding processing module decodes the video encoded frames in the buffer queue sequentially to obtain the original video frame corresponding to each video encoded frame. The mosaic processing module performs mosaic processing on the original video frame corresponding to each video encoded frame cached in the buffer queue, based on the sensitive information carried by the video encoded frame or the sensitive information carried by the associated video frame in the buffer queue that matches the video encoded frame, to obtain a mosaic video. The rendering and playback module renders and plays the mosaic video.

[0006] In one possible implementation, the mosaic processing module is further configured to perform image stabilization processing on the original video frame corresponding to the video encoded frame when the video encoded frame does not carry sensitive information.

[0007] In one possible implementation, the device includes a video acquisition module, a video encoding module, a sensitive information detection module, and a video frame preprocessing module, wherein: the video acquisition module is used to acquire video to obtain a video stream; the video encoding module is used to encode the video frames in the video stream to obtain encoded video frames; the sensitive information detection module is used to detect sensitive information in the video frames in the video stream to obtain sensitive information of the video frames; and the video frame preprocessing module is used to add the sensitive information of the video frames to the protocol header of the encoded video frames to generate the first video file.

[0008] In one possible implementation, the server includes a data management module and a media service module, wherein: the data management module is used to receive a first video file uploaded by the device and save the first video file to a storage space; the media service module is used to retrieve the first video file from the storage space according to the user's video playback request, encapsulate the first video file in FLV format to obtain an encapsulated second video file, and transmit the second video file to the browser.

[0009] In one possible implementation, the server further includes a device management module, wherein the device management module is used to authenticate the device and establish a signaling link with the device to confirm the online status of the device.

[0010] A second aspect of this application provides a mosaic video playback method, which is applied to the browser side of the mosaic video playback control system as described in the first aspect, comprising: retrieving a second video file from a server; decapsulating the second video file to obtain video encoded frames in the second video file; decoding each video encoded frame in the second video file to obtain the original video frame corresponding to the video encoded frame; performing mosaic processing on the original video frame corresponding to the video encoded frame according to sensitive information carried by the video encoded frame or sensitive information carried by an associated video frame matching the video encoded frame to obtain a mosaic video; and calling a WebGL component to render and play the mosaic video.

[0011] In one possible implementation, the step of performing mosaic processing on the original video frame corresponding to the video encoded frame based on sensitive information carried by the video encoded frame or sensitive information carried by an associated video frame matching the video encoded frame to obtain a mosaic video includes: determining, according to the timestamp order of the video encoded frames, whether each video encoded frame in the second video file carries sensitive information; if the video encoded frame carries sensitive information, determining the location of the region to be mosaic processed based on the sensitive information carried by the video encoded frame, and performing mosaic processing on the original video frame corresponding to the video encoded frame according to the location of the region; if the video encoded frame does not carry sensitive information, determining the location of the region to be mosaic processed based on the sensitive information carried by an associated video frame matching the video encoded frame, and performing mosaic processing on the original video frame corresponding to the video encoded frame according to the location of the region.

[0012] In one possible implementation, the step of determining the region location requiring mosaic processing based on sensitive information carried by an associated video frame matching the video encoded frame, and performing mosaic processing on the original video frame corresponding to the video encoded frame according to the region location, includes: calculating the time difference between the timestamp of the video encoded frame and the timestamp of the associated video frame matching the video encoded frame; determining whether the time difference exceeds a preset threshold; if the time difference does not exceed the preset time threshold, then determining the region location requiring mosaic processing based on the sensitive information carried by the associated video frame matching the video encoded frame, and performing mosaic processing on the original video frame corresponding to the video encoded frame according to the region location.

[0013] A third aspect of this application provides a mosaic video playback method, which is applied to the server side of the mosaic video playback control system as described in the first aspect, comprising: receiving a user's video playback request, the video playback request carrying the user's identity information; determining the user's permissions based on the user's identity information, wherein the user's permissions include ordinary user permissions and administrator user permissions; obtaining a first video file matching the video playback request from storage space; if the user's permissions are administrator user permissions, directly encapsulating the first video file in FLV format; if the user's permissions are ordinary user permissions, extending the video encoding frame in the first video file with SEI NALU units and adding sensitive information of the video encoding frame to the body of the SEI NALU unit, encapsulating the video encoding frame with extended SEI NALU units in FLV format to obtain a second encapsulated video file; and transmitting the second video file to the browser in response to the user's video playback request.

[0014] A fourth aspect of this application provides a mosaic video playback method, which is applied to the device side of the mosaic video playback control system as described in the first aspect, comprising: performing video acquisition while the device is powered on to obtain a video stream; performing encoding processing and sensitive information detection processing on video frames in the video stream to obtain a video encoded frame corresponding to the video frame and sensitive information in the video frame; and adding the sensitive information in the video frame to the protocol header of the video encoded frame to generate a first video file.

[0015] A fifth aspect of this application provides an electronic device including a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that the processor, when executing the computer program, implements the steps of the method as described in any one of the second, third, and fourth aspects.

[0016] A sixth aspect of this application provides a computer-readable storage medium storing a computer program, characterized in that, when executed by a processor, the computer program implements the steps of the method as described in any one of the second, third, and fourth aspects.

[0017] A seventh aspect of this application provides a computer program product that, when run on an electronic device, causes the electronic device to execute the steps of the method as described in any one of the second, third, and fourth aspects.

[0018] The beneficial effects of the embodiments in this application compared with the prior art are:

[0019] This application uses a device to collect video data and encode and detect sensitive information in the video frames. This process yields the encoded frame and sensitive information within the video frame. The sensitive information is added to the protocol header of the encoded frame to obtain a first video file, which is then uploaded to a server. The server, based on the user's video playback request, encapsulates the first video file to obtain a second video file, which is then transmitted to a browser. The browser receives the second video file, performs mosaic processing on its video frames to obtain a mosaic video, and renders and plays the mosaic video. This reduces server storage costs, increases server video concurrency, improves video playback response speed, and supports plugin-free dynamic playback of mosaic videos. Attached Figure Description

[0020] To more clearly illustrate the technical solutions in the embodiments of this application, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0021] Figure 1 A schematic diagram of the basic structure of a mosaic video playback control system provided in this application embodiment;

[0022] Figure 2 A schematic diagram of the browser side in a mosaic video playback control system provided in an embodiment of this application;

[0023] Figure 3 A schematic diagram of the device side in the mosaic video playback control system provided in this application embodiment;

[0024] Figure 4 A schematic diagram of the server-side structure in a mosaic video playback control system provided in an embodiment of this application;

[0025] Figure 5 A flowchart illustrating the implementation of a mosaic video playback method applied in a browser, as provided in this application embodiment;

[0026] Figure 6 A flowchart illustrating an implementation of a mosaic processing method for playing mosaic videos on a browser side, as provided in this application embodiment;

[0027] Figure 7 Another implementation flowchart of mosaic processing in the mosaic video playback method on the browser side provided in the embodiments of this application;

[0028] Figure 8 A schematic diagram of a video frame sequence provided in an embodiment of this application;

[0029] Figure 9 A flowchart illustrating the implementation of a mosaic video playback method applied on a server side, as provided in this application embodiment;

[0030] Figure 10 A flowchart illustrating the implementation of a mosaic video playback method applied on a device, as provided in an embodiment of this application;

[0031] Figure 11 This is a basic structural block diagram of an electronic device provided in an embodiment of this application. Detailed Implementation

[0032] In the following description, specific details such as particular system architectures and techniques are set forth for illustrative purposes and not for limitation, in order to provide a thorough understanding of the embodiments of this application. However, those skilled in the art will understand that this application may also be implemented in other embodiments without these specific details. In other instances, detailed descriptions of well-known systems, apparatuses, circuits, and methods have been omitted so as not to obscure the description of this application with unnecessary detail.

[0033] It should be understood that, when used in this application specification and the appended claims, the term "comprising" indicates the presence of the described features, integrals, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or a collection thereof.

[0034] It should also be understood that the term “and / or” as used in this application specification and the appended claims means any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.

[0035] As used in this application specification and the appended claims, the term "if" may be interpreted, depending on the context, as "when," "once," "in response to determination," or "in response to detection." Similarly, the phrase "if determined" or "if detected [the described condition or event]" may be interpreted, depending on the context, as meaning "once determined," "in response to determination," "once detected [the described condition or event]," or "in response to detection [the described condition or event]."

[0036] Furthermore, in the description of this application and the appended claims, the terms "first," "second," "third," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0037] References to "one embodiment" or "some embodiments" in this specification mean that one or more embodiments of this application include a specific feature, structure, or characteristic described in connection with that embodiment. Therefore, the phrases "in one embodiment," "in some embodiments," "in other embodiments," "in still other embodiments," etc., appearing in different parts of this specification do not necessarily refer to the same embodiment, but rather mean "one or more, but not all, embodiments," unless otherwise specifically emphasized. The terms "comprising," "including," "having," and variations thereof mean "including but not limited to," unless otherwise specifically emphasized. "A plurality" means "two or more."

[0038] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0039] In some embodiments of this application, please refer to Figure 1 , Figure 1 This application provides a schematic diagram of the basic structure of a mosaic video playback control system, as shown in the embodiments. Figure 1 As shown, the mosaic video playback control system mainly includes a device terminal 10, a server terminal 20, and a browser terminal 30. Through the interaction of the three terminals, the video stream can be rendered and played while being mosaicked, thus realizing dynamic mosaic video playback.

[0040] In this embodiment, device 10 is used to collect video data and perform encoding processing and sensitive information detection on video frames in the video data, obtaining the encoded frame corresponding to the video frame and the sensitive information in the video frame. The sensitive information is added to the protocol header of the encoded frame to obtain a first video file, which is then uploaded to the server. In one specific embodiment, encoding processing and sensitive information detection processing can be performed on each video frame. For each video frame, the sensitive information detected from the video frame is added to the protocol header of the encoded frame corresponding to that video frame. In another specific implementation, sensitive information detection can be performed on video frames in the video data according to a frame skipping strategy. According to the time when the sensitive information is obtained, the currently detected sensitive information is added to the protocol header of the currently encoded frame. In this embodiment, sensitive information such as faces and license plate numbers in the video frame can be obtained through sensitive information detection. The sensitive information includes the location information of the faces and license plate numbers in the video frame, including horizontal coordinates, vertical coordinates, width, and height. By adding the sensitive information to the protocol header of the encoded frame, the additional storage space occupied is very small, which can save storage costs.

[0041] In this embodiment, server 20 is used to encapsulate the first video file uploaded by device 10 according to the user's video playback request, obtain the encapsulated second video file, and transmit the second video file to browser 30. Specifically, after obtaining the first video file uploaded by device 10, server 20 can directly store the first video file in its storage space. When the user needs to play the video file uploaded by device 10, they can request playback in browser 30, and browser 30 will send the user's video playback request to server 20. After receiving the user's video playback request, server 20 can obtain the first video file matching the video playback request from its storage space, and then encapsulate the video frames in the first video file into FLV format to obtain the encapsulated second video file, which is then transmitted to browser 30. It is understood that FLV is a streaming media video format that developed with the introduction of Flash MX, resulting in small file sizes and fast loading speeds. It should be noted that the video frames in the second video file are pre-extended with an SEI NALU unit. The NALU unit records sensitive information carried by the video frame, without disrupting the original H.264 / H.265 video frame structure. This allows for the distribution of audio and video data using standard streaming media transmission protocols such as HTTP-FLV, HLS, and RTMP, without compromising the standardization of streaming media transmission protocols, demonstrating good versatility and scalability. SEI (Supplemental Enhancement Information) provides a method for adding extra information to the video stream and is a feature of the H.264 / H.265 video compression standard. NALU (Network Abstraction Layer Unit) is used to encapsulate data provided by the video coding layer for network transmission. HTTP-FLV (Flash Video Over HTTP) refers to a transmission protocol for FLV audio and video data based on HTTP. HLS (HTTP Live Streaming) refers to a streaming media network transmission protocol based on HTTP. RTMP (Real Time Messaging Protocol) is an open protocol for direct audio, video, and data transmission between Flash players and servers. In the mosaic video playback control system, server 20 only needs to provide FLV format encapsulation and video data forwarding functions, consuming minimal CPU (Central Processing Unit) and GPU (Graphics Processing Unit) resources, thus increasing the video concurrency of server 20.Since the device 10 has already identified the sensitive information and attached it to the protocol header of the video frame, the server 20 does not need to identify the information. It only needs to encapsulate it in FLV format and transmit it to the browser 30, which can improve the response speed of video playback.

[0042] In this embodiment, the browser client 30 receives a second video file transmitted by the server client 20, performs mosaic processing on the video frames in the second video file to obtain a mosaic video, and renders and plays the mosaic video. Specifically, the browser client 30 acts as a video playback client, enabling human-computer interaction with the user. The user can initiate a video playback request by performing relevant operations in the interactive interface of the browser client 30. The browser client 30 sends the video playback request to the server client 20 to retrieve the video file to be played, which is the second video file encapsulated by the server client 20. After obtaining the second video file, the browser client 30 decapsulates the second video file to obtain the video frames in the second video file. The obtained video frames are encoded video frames. By decoding the encoded video frames, the original H264 / H265 video frames are obtained. By parsing the SEINALU unit extended before the encoded video frames, sensitive information recorded in the SEINALU unit can be obtained. Then, by performing mosaic processing on the original video frames based on the sensitive information, the mosaic video can be obtained. Furthermore, by calling the WebGL component on the browser side 30, the mosaic video can be rendered and played. WebGL (Web Graphics Library) is a 3D graphics protocol that provides hardware-accelerated 3D rendering. Based on the above embodiment, the mosaic video playback control system moves the detection of faces and license plates in the video from the server side 20 to the device side 10, and the video mosaic processing from the server side 20 to the browser side 30. Utilizing edge computing and browser front-end processing, video processing is avoided on the server side 20. The server side 20 only needs to provide FLV format encapsulation and data forwarding, consuming very few CPU and GPU resources, and significantly improving video concurrency.

[0043] In some embodiments of this application, please refer to Figure 2 , Figure 2 This is a schematic diagram of the browser side of the mosaic video playback control system provided in an embodiment of this application. Figure 2As shown, the browser side 30 includes a JS layer 21 and a WASM layer 22. Specifically, the plugin-free playback library of the H5 page in the browser is physically a JS library, which is logically divided into a JS layer 21 and a WASM layer 22. The JS layer 21 is written in JavaScript, and the WASM layer 22 is compiled using WebAssembly technology. In this embodiment, the JS layer 21 is configured with a video streaming module 211 and a rendering and playback module 212. Specifically, the video streaming module 211 is used to pull a second video file from the server 20. The rendering and playback module 212 is used to render and play the mosaic video. The WASM layer 22 is configured with a decapsulation processing module 221, a video frame buffer module 222, a decoding processing module 223, and a mosaic processing module 224. Specifically, the decapsulation processing module 221 is used to decapsulate the second video file to obtain multiple video encoded frames. It should be noted that all or part of these multiple video encoded frames carry sensitive information. The video frame buffering module 222 is used to sequentially cache the multiple video encoded frames into a buffer queue according to their respective timestamps. The decoding processing module 223 is used to decode the video encoded frames in the buffer queue in sequence to obtain the original video frame corresponding to each video encoded frame. The mosaic processing module 224 is used to perform mosaic processing on the original video frame corresponding to each video encoded frame cached in the buffer queue, based on the sensitive information carried by the video encoded frame or the sensitive information carried by the associated video frame in the buffer queue that matches the video encoded frame, to obtain a mosaic video. In this embodiment, the browser 30 uses WebAssembly technology to compile the C / C++ program, including video decapsulation, decoding, mosaic processing, and video frame buffering, into a wasm layer 22, which has high security and can maximize user privacy security. By calling the JS layer 21 to render and play the mosaic video, both performance and security requirements can be met. The WASM layer 22 can be published with the web application without the need to install other plugins on the browser side 30, supporting plugin-free dynamic mosaic video playback on the browser side 30.

[0044] In some embodiments of this application, please refer to Figure 3 , Figure 3 This is a schematic diagram of the device side of a mosaic video playback control system provided in an embodiment of this application. Figure 3As shown, the device 10 is equipped with a video acquisition module 31, a video encoding module 32, a sensitive information detection module 33, and a video frame preprocessing module 34. In this embodiment, the video acquisition module 31 is used to acquire video data and obtain a video stream. The video encoding module 32 is used to encode the video frames in the video stream to obtain encoded video frames. The sensitive information detection module 33 is used to detect sensitive information in the video frames in the video stream to obtain the sensitive information of the video frames. The video frame preprocessing module 34 is used to add the sensitive information of the video frames to the protocol header of the encoded video frames to generate a first video file. Specifically, the device 10, as the producer of the video file, can control the camera to acquire video data and obtain a video stream through the video acquisition module. In this embodiment, the video stream is represented as a sequence of video frames, and the video frames in the sequence are ordered according to their timestamps. After the device 10 obtains the video stream, the video encoding module 32 sequentially performs H.264 / H.265 video encoding on each video frame in the video stream according to its timestamp, obtaining the video encoded frame corresponding to each video frame. Simultaneously, the sensitive information detection module 33 performs sensitive information detection on some or all video frames in the video stream, obtaining sensitive information from some or all video frames. In this embodiment, the video encoding module 32 and the sensitive information detection module 33 are executed simultaneously by two independent threads. In this embodiment, the video encoding module 32 sequentially encodes each video frame in the video stream, while to save CPU and GPU resources on the device 10, the sensitive information detection module 33 performs sensitive information detection on the video frames in the video stream according to a pre-defined frame skipping identification strategy, i.e., only obtaining sensitive information from a portion of the video frames. For example, the video encoding module 32 encodes video frames at a frame rate of 25 frames per second, while the sensitive information detection module 33 identifies 10 to 16 frames per second. In this embodiment, when the sensitive information detection module 33 detects sensitive information, in order to avoid affecting the real-time performance of the video, the video frame preprocessing module 34 adds the sensitive information of the video frame currently detected by the sensitive information detection module and the timestamp of the detected video frame to the protocol header of the encoded video frame currently encoded by the video encoding module.

[0045] In some embodiments of this application, since the video encoding module 32 and the sensitive information detection module 33 of the device 10 are executed simultaneously by two independent threads, and the sensitive information detection module 33 performs frame skipping identification, the sensitive information carried in the protocol header of the current video frame in the first video file belongs to a historical video frame. That is, the first video file may have some video frames carrying sensitive information and some video frames not carrying sensitive information. Based on this, in this embodiment, the mosaic processing module 224 in the browser 30 is also used to perform image stabilization processing on the original video frame corresponding to the video encoding frame when the video encoding frame does not carry sensitive information. Specifically, the logic of the image stabilization process is as follows: If the first video frame does not carry sensitive information, the time difference between the timestamp of the first video frame and the timestamp of the associated video frame that matches the first video frame is calculated. It is then determined whether the time difference exceeds a preset threshold. If the time difference does not exceed the preset threshold, the area represented by the sensitive information carried by the associated video frame is used to perform mosaic processing on the first video frame. If the time difference exceeds the preset threshold, the first video frame is not mosaic processed. Similarly, if a non-first video frame does not carry sensitive information, the time difference between the timestamp of the non-first video frame and the timestamp of the associated video frame that matches the first video frame is calculated. It is then determined whether the time difference exceeds a preset threshold. If the time difference does not exceed the preset threshold, the area represented by the sensitive information carried by the associated video frame is used to perform mosaic processing on the non-first video frame. If the time difference exceeds the preset threshold, the non-first video frame is not mosaic processed. It is understood that in this embodiment, whether it is the first video frame or a non-first video frame, when the video frame carries sensitive information, the video frame is directly mosaicked according to the region location represented by the sensitive information.

[0046] In some embodiments of this application, please refer to Figure 4 , Figure 4 This is a schematic diagram of the server-side structure in the mosaic video playback control system provided in an embodiment of this application. Figure 4As shown, the server 20 is configured with a data management module 41 and a media service module 42. In this embodiment, the data management module 41 is used to receive the first video file uploaded by the device and save the first video file to the storage space. The media service module 42 is used to retrieve the first video file from the database according to the user's video playback request, encapsulate the first video file in FLV format, and transmit the encapsulated second video file to the browser. Specifically, when encapsulating the first video file in FLV format, the media service module 42 can confirm the user's permissions based on the user's video playback request. If the user is confirmed to have ordinary user permissions, an SEI NALU unit is extended before the video frame, sensitive information is added to the body of the NALU unit, and then FLV format encapsulation is performed; if the user is confirmed to have administrator user permissions, the video frame is directly encapsulated in FLV format.

[0047] In some embodiments of this application, the server 20 is further configured with a device management module. This module is used to authenticate the device and establish a signaling link with it to confirm its online status. In this embodiment, after startup, the device 10 automatically connects to the server 20 and requests to establish a signaling link for communication between them. The server 20 authenticates the requesting device 10 through the device management module. Once authentication is successful, a signaling link is established between the device 10 and the server 20, setting the device 10 to an online state. At this time, the device 10 can proactively report the generated first video file to the server 20.

[0048] In some embodiments of this application, please refer to Figure 5 , Figure 5 This is a flowchart illustrating the implementation of a mosaic video playback method applied in a browser, as provided in an embodiment of this application. Figure 5 As shown, it may specifically include steps S51 to S55.

[0049] S51: Retrieve the second video file from the server;

[0050] S52: Decapsulate the second video file to obtain the video encoded frames in the second video file;

[0051] S53: For each video encoded frame in the second video file, the video encoded frame is decoded to obtain the original video frame corresponding to the video encoded frame;

[0052] S54: Based on the sensitive information carried by the video encoded frame or the sensitive information carried by the associated video frame that matches the video encoded frame, perform mosaic processing on the original video frame corresponding to the video encoded frame to obtain a mosaic video.

[0053] S55: Call the WebGL component to render and play the mosaic video.

[0054] In this embodiment, on the browser side, the user can initiate a video playback request by inputting operations on the browser's interactive page, thereby requesting the playback of the desired video file. After the user initiates the video playback request, the browser sends the video playback request to the server, and retrieves a second video file matching the video playback request from the server via streaming media transmission protocols such as HTTP-FLV, HLS, and RTMP. Since the second video file is obtained by encoding the video stream on the device side and encapsulating it in FLV format on the server side, the browser needs to decapsulate the second video file after obtaining it from the server to obtain all the video encoded frames in the second video file. Then, it needs to decode each video encoded frame in the second video file to obtain the original video frame corresponding to each video encoded frame. Furthermore, by performing SEI parsing on each video encoded frame in the second video file, it is determined whether each video encoded frame carries sensitive information. For video encoded frames carrying sensitive information, the sensitive information in the video encoded frame is obtained. It should be noted that the sensitive information in the video encoded frame is represented as region location information, including the horizontal coordinate, vertical coordinate, width, and height of the region. For example, information such as the x-coordinate, y-coordinate, width, and height of the area containing a face in a video frame, or the x-coordinate, y-coordinate, width, and height of the area containing a license plate in the video. After obtaining the original video frames corresponding to each video encoded frame in the second video file, along with the sensitive information they carry, the browser can sequentially perform mosaic processing on the original video frames corresponding to each video encoded frame according to the timestamp order, based on the sensitive information carried by the video encoded frames or the sensitive information carried by associated video frames that match the video encoded frames, thus obtaining a mosaic video. Then, the WebGL component in the browser's JavaScript library is called to render and play the obtained mosaic video, allowing the video stream to be rendered and played simultaneously while being mosaic processed on the browser.

[0055] It is understood that in this embodiment, if none of the video encoded frames in the second video file carry sensitive information, there is no need to perform mosaic processing on the video encoded frames. The video encoded frames can be decoded directly according to the timestamp order of each video encoded frame before being rendered and played.

[0056] In some embodiments of this application, please refer to Figure 6 , Figure 6 This is a flowchart illustrating an implementation of a mosaic video playback method in a browser, as provided in this application embodiment, for mosaic processing. Figure 6As shown, it may specifically include steps S61 to S63.

[0057] S61: According to the timestamp order of the video encoded frames, for each video encoded frame in the second video file, determine whether the video encoded frame carries sensitive information;

[0058] S62: If the video encoded frame carries sensitive information, determine the location of the area that needs to be mosaicked according to the sensitive information carried by the video encoded frame, and perform mosaic processing on the original video frame corresponding to the video encoded frame according to the location of the area.

[0059] S63: If the video encoded frame does not carry sensitive information, then based on the sensitive information carried by the associated video frame that matches the video encoded frame, determine the location of the area that needs to be mosaicked, and perform mosaicking on the original video frame corresponding to the video encoded frame according to the location of the area.

[0060] In this embodiment, to conserve CPU and GPU resources on the device side, a frame skipping detection strategy is used to detect sensitive information in the video stream. Specifically, only a portion of the video encoded frames in the second video file obtained by the browser will carry sensitive information. In this embodiment, the browser performs mosaic processing frame by frame according to the timestamp order of the video encoded frames. For each video encoded frame, it is first determined whether the video encoded frame carries sensitive information. Specifically, after obtaining the video encoded frame through decapsulation on the browser side, SEI parsing is further used to identify the extended SEI NALU units in the video encoded frame. If the body of the SEI NALU unit records the sensitive information of the video encoded frame, it indicates that the video encoded frame carries sensitive information. If no extended SEI NALU unit is parsed through SEI parsing, it indicates that the video encoded frame does not carry sensitive information. If the video encoded frame carries sensitive information, the location of the area in the video encoded frame that needs mosaic processing can be directly determined based on the sensitive information, and mosaic processing is performed on the corresponding original video frame according to the location of the area. If the video encoded frame does not carry sensitive information, the location of the area in the video encoded frame that needs to be mosaicked can be determined based on the sensitive information carried by the associated video frame that matches the video encoded frame, and the original video frame corresponding to the video encoded frame can be mosaicked according to the location of the area.

[0061] In some embodiments of this application, please refer to Figure 7 , Figure 7 This is another implementation flowchart of the mosaic processing method in the browser-side mosaic video playback method provided in the embodiments of this application. For example... Figure 7 As shown, it may specifically include steps S71 to S72.

[0062] S71: Calculate the time difference between the timestamp of the video encoded frame and the timestamp of the associated video frame that matches the video encoded frame;

[0063] S72: Determine whether the time difference exceeds a preset threshold. If the time difference does not exceed the preset time threshold, determine the location of the area to be mosaicked based on the sensitive information carried by the associated video frame that matches the video encoded frame, and perform mosaic processing on the original video frame corresponding to the video encoded frame according to the location of the area.

[0064] Please refer to the following in this embodiment: Figure 8 , Figure 8 This is a schematic diagram of a video frame sequence provided in an embodiment of this application. In this embodiment, the browser can set up a video frame buffer queue to cache video encoded frames decapsulated from the second video file in chronological order of timestamps. If the video encoded frame carries sensitive information, the sensitive information is bound to the video encoded frame and cached together in the buffer queue. After receiving the video encoded frame, the browser needs to buffer a certain time value, which is a preset threshold set by the user. If the time is lower than this preset threshold, playback needs to be waited for. In this embodiment, during the mosaic processing of the video encoded frame, if the video encoded frame does not carry sensitive information, the time difference between the timestamp of the video encoded frame and the timestamp of the associated video frame that matches the video encoded frame is calculated. By judging whether the time difference exceeds the preset threshold, if the time difference does not exceed the preset threshold, the location of the area to be mosaic processed is determined according to the sensitive information carried by the associated video frame that matches the video encoded frame, and then the original video frame corresponding to the video encoded frame is mosaic processed according to the location of the area. For example, as shown... Figure 8As shown, on the browser side, received video encoded frames are sorted according to their timestamps. Assuming the timestamp of the first video encoded frame is denoted as pts1, the timestamp of the previous video encoded frame carrying sensitive information is determined to be the associated video encoded frame matching the first frame, denoted as pts2, and the timestamps of non-first frames are denoted as pts3. The preset threshold is set to 200 milliseconds. If the currently processed video frame is the first frame, the timestamp pts1 of the first video encoded frame is subtracted from the timestamp pts2 of the previous video encoded frame carrying sensitive information to obtain the time difference. When pts1-pts2≤200ms, the mosaic region of the previous video encoded frame carrying sensitive information is used to mosaic the first video encoded frame. When pts1-pts2>200ms, the first video encoded frame is not mosaicked. If the currently encoded video frame undergoing mosaic processing is not the first frame, the timestamp pts3 of the non-first frame is subtracted from the timestamp pts2 of the previous encoded video frame carrying sensitive information to obtain the time difference. When pts3-pts2≤200ms, the mosaic region of the previous encoded video frame carrying sensitive information is used to mosaic the non-first frame. When pts1-pts2>200ms, no mosaic is performed on the non-first frame.

[0065] In some embodiments of this application, please refer to Figure 9 , Figure 9 This is a flowchart illustrating the implementation of a mosaic video playback method applied on a server side, as provided in an embodiment of this application. Figure 9 As shown, it may specifically include steps S91 to S94.

[0066] S91: Receive a user's video playback request, wherein the video playback request carries the user's identity information;

[0067] S92: Determine the user's permissions based on the user's identity information, wherein the user's permissions include ordinary user permissions and administrator user permissions;

[0068] S93: Obtain a first video file matching the video playback request from the storage space. If the user has administrator privileges, directly encapsulate the first video file in FLV format. If the user has regular user privileges, extend the SEI NALU unit for the video encoded frame in the first video file and add the sensitive information of the video encoded frame to the SEI NALU unit body. Encapsulate the video encoded frame with extended SEI NALU unit in FLV format to obtain the encapsulated second video file.

[0069] S94: In response to the user's video playback request, the second video file is transmitted to the browser.

[0070] In this embodiment, the need for video playback filtering can be determined based on the user's permissions. Specifically, the user initiates a video playback request by performing relevant operations in the browser's interactive interface. The video request carries the user's identity information and the video source information to be played. The browser then sends the video playback request to the server. The user's identity information can be account ID, identity ID, etc., and the video source information includes the video's source device information and time range information. After receiving the user's video playback request, the server can determine the user's permissions based on the user's identity information. Specifically, user permissions include ordinary user permissions and administrator user permissions. The server stores a list of administrator users, which records the identity information of all administrator users. In this embodiment, the list of administrator users can be traversed based on the user's identity information to confirm whether the user is an administrator user. If the user is an administrator user, their permissions are determined to be administrator user permissions; otherwise, their permissions are determined to be ordinary user permissions. After determining the user's permissions, the server can retrieve the first video file matching the video playback request from its storage space based on the video source information carried in the video request. When encapsulating the first video file into FLV format, if the user has administrator privileges, the first video file is directly encapsulated into FLV format. If the user has regular user privileges, the SEI NALU unit is extended for the video encoded frames in the first video file, and sensitive information of the video encoded frames is added to the body of the SEI NALU unit. Then, the video encoded frames with extended SEI NALU units are encapsulated into FLV format. The encapsulated second video file is obtained through FLV format encapsulation. After the server encapsulates the second video file, it can respond to the user's video playback request and transmit the second video file to the browser using standard streaming media transmission protocols such as HTTP-FLV, HLS, and RTMP.

[0071] In some embodiments of this application, please refer to Figure 10 , Figure 10 This document provides a flowchart illustrating the implementation of a mosaic video playback method applied on a device, as an embodiment of this application. Figure 10 As shown, it may specifically include steps S101 to S103.

[0072] S101: Video capture is performed while the device is powered on to obtain a video stream;

[0073] S102: Encode and detect sensitive information for each video frame in the video stream to obtain the video encoded frame corresponding to the video frame and the sensitive information in the video frame;

[0074] S103: Add the sensitive information in the video frame to the protocol header of the video encoded frame to generate the first video file.

[0075] In this embodiment, the device acts as the producer of the video file. After the device is powered on, it acquires video data within its visual range in real time through the camera, obtaining a video stream. The video frames in this stream are in YUV format. The device is configured with two threads. One thread performs H264 / H265 encoding on each acquired YUV video frame to obtain the corresponding encoded video frames. The other thread performs sensitive information detection processing on the acquired YUV video frames to obtain sensitive information within them. In some specific implementations, a frame skipping detection strategy can be used to perform sensitive information detection processing on the acquired YUV video frames. When the current video frame is detected to contain sensitive information such as faces and license plates, the detected sensitive information is added to the video frame protocol header, thereby generating the first video file.

[0076] It is understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0077] In some embodiments of this application, please refer to Figure 11 , Figure 11 This is a basic structural block diagram of an electronic device provided in an embodiment of this application. Figure 11 As shown, the electronic device 11 of this embodiment includes: a processor 111, a memory 112, and a computer program 113 stored in the memory 112 and executable on the processor 111, such as the programs for the various mosaic video playback methods described above. When the processor 111 executes the computer program 113, it implements the steps in the various embodiments of the mosaic video playback methods described above. Alternatively, when the processor 111 executes the computer program 113, it implements the functions of each module in the embodiments corresponding to the mosaic video playback control system described above. Please refer to the relevant descriptions in the embodiments for details, which will not be repeated here.

[0078] For example, the computer program 113 can be divided into one or more modules (units), which are stored in the memory 112 and executed by the processor 111 to complete this application. The one or more modules can be a series of computer program instruction segments capable of performing specific functions, which describe the execution process of the computer program 113 in the electronic device 11. For example, the computer program 113 can be divided into various modules in the aforementioned mosaic video playback control system, each module having the function described above.

[0079] The electronic device may include, but is not limited to, a processor 111 and a memory 112. Those skilled in the art will understand that... Figure 11 This is merely an example of electronic device 11 and does not constitute a limitation on electronic device 11. It may include more or fewer components than shown, or combine certain components, or different components. For example, the electronic device may also include input / output devices, network access devices, buses, etc.

[0080] The processor 111 can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor.

[0081] The memory 112 can be an internal storage unit of the electronic device 11, such as a hard disk or RAM of the electronic device 11. The memory 112 can also be an external storage device of the electronic device 11, such as a plug-in hard disk, Smart Media Card (SMC), Secure Digital (SD) card, or Flash Card equipped on the electronic device 11. Furthermore, the memory 112 can include both internal and external storage units of the electronic device 11. The memory 112 is used to store the computer program and other programs and data required by the electronic device. The memory 112 can also be used to temporarily store data that has been output or will be output.

[0082] It should be noted that the information interaction and execution process between the above modules / units are based on the same concept as the method embodiments of this application. For details on their specific functions and technical effects, please refer to the method embodiments section, and they will not be repeated here.

[0083] This application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, can implement the steps in the various method embodiments described above. In this embodiment, the computer-readable storage medium can be either non-volatile or volatile.

[0084] This application provides a computer program product that, when run on a mobile terminal, enables the mobile terminal to implement the steps described in the various method embodiments.

[0085] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is merely an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the system can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit. Furthermore, the specific names of the functional units and modules are only for easy differentiation and are not intended to limit the scope of protection of this application. The specific working process of the units and modules in the above system can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0086] If the integrated module / unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments can also be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include: any entity or device capable of carrying the computer program code, recording media, USB flash drives, portable hard drives, magnetic disks, optical disks, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc. It should be noted that the content included in the computer-readable medium can be appropriately added or removed according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable media do not include electrical carrier signals and telecommunication signals.

[0087] In the above embodiments, the descriptions of each embodiment have different focuses. For parts that are not described in detail or recorded in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0088] The above-described embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be included within the protection scope of this application.

Claims

1. A mosaic video playback control system, characterized in that, The system includes a device, a server, and a browser, wherein: The device is used to collect video data and encode and detect sensitive information in the video frames in the video data respectively, to obtain the encoded frame corresponding to the video frame and the sensitive information in the video frame, to add the sensitive information to the protocol header of the encoded frame, to obtain a first video file, and to upload the first video file to the server. The server is used to encapsulate the first video file uploaded by the device according to the user's video playback request, and transmit the encapsulated second video file to the browser. The browser is used to receive the second video file transmitted by the server, perform mosaic processing on the video frames in the second video file to obtain a mosaic video, and render and play the mosaic video. The server is also used for: Receive a user's video playback request, wherein the video playback request carries the user's identity information; Based on the user's identity information, the user's permissions are determined, wherein the user's permissions include ordinary user permissions and administrator user permissions; The system retrieves a first video file matching the video playback request from the storage space. If the user has administrator privileges, the first video file is directly encapsulated in FLV format. If the user has regular user privileges, the system extends the video encoding frames in the first video file with SEI NALU units and adds sensitive information of the video encoding frames to the body of the SEI NALU units. The video encoding frames with extended SEI NALU units are then encapsulated in FLV format to obtain the encapsulated second video file. In response to the user's video playback request, the second video file is transmitted to the browser.

2. The mosaic video playback control system according to claim 1, characterized in that, The browser side includes a JS layer and a WASM layer. The JS layer is configured with a video streaming module and a rendering and playback module. The WASM layer is configured with a decapsulation processing module, a video frame buffer module, a decoding processing module, and a mosaic processing module, wherein: The video streaming module is used to retrieve a second video file from the server. The decapsulation processing module is used to decapsulate the second video file to obtain multiple video encoded frames; The video frame buffer module is used to cache the multiple video encoded frames sequentially into a buffer queue according to the timestamps corresponding to each of the multiple video encoded frames. The decoding processing module is used to decode the video encoded frames in the buffer queue in sequence to obtain the original video frame corresponding to each video encoded frame. The mosaic processing module is used to perform mosaic processing on the original video frame corresponding to each video encoded frame cached in the buffer queue, based on the sensitive information carried by the video encoded frame or the sensitive information carried by the associated video frame in the buffer queue that matches the video encoded frame, to obtain a mosaic video. The rendering and playback module is used to render and play the mosaic video.

3. The mosaic video playback control system according to claim 2, characterized in that, The mosaic processing module is also used to perform image stabilization processing on the original video frame corresponding to the video encoding frame when the video encoding frame does not carry sensitive information.

4. The mosaic video playback control system according to claim 1, characterized in that, The device includes a video acquisition module, a video encoding module, a sensitive information detection module, and a video frame preprocessing module, wherein: The video acquisition module is used to acquire video and obtain a video stream; The video encoding module is used to encode the video frames in the video stream to obtain video encoded frames; The sensitive information detection module is used to detect sensitive information in video frames in the video stream and obtain the sensitive information of the video frames. The video frame preprocessing module is used to add sensitive information of the video frame to the protocol header of the video encoded frame to generate the first video file.

5. The mosaic video playback control system according to claim 1, characterized in that, The server-side includes a data management module and a media service module, wherein: The data management module is used to receive the first video file uploaded by the device and save the first video file to the storage space; The media service module is used to retrieve the first video file from the storage space according to the user's video playback request, encapsulate the first video file in FLV format to obtain the encapsulated second video file, and transmit the second video file to the browser.

6. The mosaic video playback control system according to claim 5, characterized in that, The server also includes a device management module, wherein: The device management module is used to authenticate the device and establish a signaling link with the device to confirm the online status of the device.

7. A method for playing mosaic video, characterized in that, The method is applied to the browser side of the mosaic video playback control system as described in any one of claims 1 to 6, including: Retrieve the second video file from the server; The second video file is decapsulated to obtain the video encoded frames from the second video file; For each video encoded frame in the second video file, the video encoded frame is decoded to obtain the original video frame corresponding to the video encoded frame; Based on the sensitive information carried by the video encoded frame or the sensitive information carried by the associated video frame that matches the video encoded frame, the original video frame corresponding to the video encoded frame is subjected to mosaic processing to obtain a mosaic video. The WebGL component is invoked to render and play the mosaic video.

8. The mosaic video playback method according to claim 7, characterized in that, The step of performing mosaic processing on the original video frame corresponding to the video encoded frame based on the sensitive information carried by the video encoded frame or the sensitive information carried by an associated video frame matching the video encoded frame to obtain a mosaic video includes: Based on the timestamp order of the video encoded frames, for each video encoded frame in the second video file, determine whether the video encoded frame carries sensitive information; If the video encoded frame carries sensitive information, the location of the area that needs to be mosaicked is determined according to the sensitive information carried by the video encoded frame, and the original video frame corresponding to the video encoded frame is mosaicked according to the location of the area. If the video encoded frame does not carry sensitive information, the location of the region that needs to be mosaicked is determined based on the sensitive information carried by the associated video frame that matches the video encoded frame, and the original video frame corresponding to the video encoded frame is mosaicked according to the location of the region.

9. The mosaic video playback method according to claim 8, characterized in that, The step of determining the location of the region requiring mosaic processing based on the sensitive information carried by the associated video frame matching the video encoded frame, and performing mosaic processing on the original video frame corresponding to the video encoded frame according to the location of the region, includes: Calculate the time difference between the timestamp of the video encoded frame and the timestamp of the associated video frame that matches the video encoded frame; If the time difference value does not exceed a preset threshold, then the location of the region that needs to be mosaicked is determined based on the sensitive information carried by the associated video frame that matches the video encoded frame, and the original video frame corresponding to the video encoded frame is mosaicked according to the location of the region.

10. A method for playing mosaic video, characterized in that, The method is applied to the device side of the mosaic video playback control system as described in any one of claims 1 to 6, including: Video capture is performed while the device is powered on to obtain a video stream. The video frames in the video stream are encoded and sensitive information detected to obtain the video encoded frame corresponding to the video frame and the sensitive information in the video frame. Sensitive information from the video frame is added to the protocol header of the video encoded frame to generate a first video file.

11. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the method as described in any one of claims 7-9 or 10.

12. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method as described in any one of claims 7-9 or 10.

Images