Video occlusion processing method, system, and video processing device

Abstract

The application discloses a video occlusion processing method, system and video processing device. The method comprises the following steps: identifying one or more target objects from a video picture, generating a corresponding layer for each target object; acquiring a target object which needs to be occluded from the one or more target objects; acquiring position information of the target object which needs to be occluded in the video picture, determining a part occupied by the target object which needs to be occluded in the video picture according to the position of the target object in the layer corresponding to the target object which needs to be occluded; and occluding the part occupied by the target object which needs to be occluded in the video picture. The application solves the problem that the target occlusion in the prior art is performed on all the same type of target, thereby failing to realize the occlusion of part of the target.

Description

Technical Field

[0001] This application relates to the field of video, and more specifically, to video occlusion processing methods, systems, and video processing equipment. Background Technology

[0002] With the development of camera technology, cameras are being used more and more, and the targets captured by cameras are becoming increasingly complex.

[0003] In existing technologies, some cameras and video analysis and processing devices are already capable of automatically detecting targets and eliminating target occlusion.

[0004] However, the inventors discovered that existing technologies for video detection and target occlusion occlude all targets in the same video frame, failing to occlude only some targets, which affects the application of video captured by the camera. Summary of the Invention

[0005] This application provides a video occlusion processing method, system, and video processing device to at least solve the problem in the prior art where target occlusion is applied to all targets of the same type, thus failing to occlude only some targets.

[0006] According to one aspect of this application, a video occlusion processing method is provided, comprising: identifying one or more target objects from a video frame; generating a corresponding layer for each target object, wherein the position of each target object in its corresponding layer is the same as its position in the video frame; obtaining a target object to be occluded from the one or more target objects, wherein the target object to be occluded is at least one; obtaining the position information of the target object to be occluded in the video frame; determining the portion occupied by the target object to be occluded in the video frame based on the position of the target object in the layer corresponding to the target object to be occluded; and occluding the portion occupied by the target object to be occluded in the video frame.

[0007] Furthermore, in each layer, a first pixel value is used to identify the target object in that layer, and a second pixel value is used to identify the rest of the layer other than the target object. The first pixel value and the second pixel value are different.

[0008] Further, the video frame and the layer are saved using video frames. The video frame includes a first part and a second part, wherein: the first part is used to save the content of the original video frame corresponding to the video frame; the second part saves the following data or index information, wherein the index information is used to indicate the saving location of the following data, the data including: a part for saving the resolution of the video frame and the resolution of the layer, a part for saving the number of target objects detected in the video frame, and at least one of the following: indication information for indicating the last target object saved in the video frame, and a part for saving the layer corresponding to the target object, wherein there are N parts for saving the layer corresponding to the target object, and each target object corresponds to one part for saving the layer corresponding to the target object, where N is the number of target objects detected in the video frame.

[0009] Furthermore, saving the layer corresponding to the target object includes: saving the pixel value of each pixel in the layer, or saving the layer as an image file; and / or, the second part also saves the coordinate information of the target object in the layer and / or in the video frame, wherein the coordinate information includes the coordinates of points on the border surrounding the target object.

[0010] Further, determining the portion of the target object to be occupied in the video frame based on the position of the target object in the layer corresponding to the target object to be occupied includes: mapping the area containing pixels with the first pixel value in the layer corresponding to the target object to be occupied to the video frame based on the resolution of the layer corresponding to the target object to be occupied and the resolution of the video frame, thereby obtaining the portion of the target object to be occupied in the video frame.

[0011] Furthermore, mapping the area containing pixels with the first pixel value in the layer corresponding to the target object to be occluded onto the video frame includes: obtaining the coordinate information of the target object, and locating the range of the target object in the video frame based on the coordinate information; wherein the coordinate information includes the coordinates of points on the border surrounding the target object; and within the range, mapping the area containing pixels with the first pixel value in the layer corresponding to the target object to be occluded onto the video frame.

[0012] Further, obtaining the target objects that need to be occluded from the one or more target objects includes at least one of the following: receiving time information and / or video channel number input by the user, and obtaining the target objects that need to be occluded from the video frames corresponding to the time information and / or video channel number; receiving a target object selected by the user from one or more target objects identified in the video frame, and using the target object selected by the user as the target object that needs to be occluded or does not need to be occluded; receiving the type or attribute of the target object input by the user, and using the target object of the type or the target object with the attribute as the target object that needs to be occluded or does not need to be occluded; receiving an image input by the user, identifying the target object from the image, and using the target object that is the same as the target object identified in the image from one or more target objects identified in the video frame as the target object that needs to be occluded or does not need to be occluded.

[0013] Furthermore, occupying the portion of the video frame occupied by the target object to be occupied includes: filling the portion of the video frame occupied by the target object to be occupied with one or more colors; and / or, dividing the portion of the video frame occupied by the target object to be occupied into multiple pixel blocks, wherein each of the multiple pixel blocks is filled with one color; and / or, obtaining a pre-saved background image corresponding to the video frame, extracting the background portion corresponding to the target object from the background image according to the position information of the target object to be occupied in the background image, and filling the extracted background portion into the portion of the video frame; wherein the background image and the video frame are images taken from the same position at the same angle, and the background image does not contain the target object.

[0014] Furthermore, dividing the portion occupied by the target object to be occupied in the video frame into multiple pixel blocks includes: dividing the occupied portion into the multiple pixel blocks; for each pixel block, obtaining the average pixel value of all pixels in that pixel block; and filling the pixel block with the average pixel value; and / or, filling the extracted background portion into the portion of the video frame includes: setting a predetermined transparency for the extracted background portion; and filling the background portion with the set transparency into the portion of the video frame.

[0015] According to another aspect of this application, a video system is also provided, comprising: a camera for capturing video images; and a video processing device connected to the camera for receiving the video images and performing the above-described method, wherein the step of identifying one or more target objects from the video images is performed by the camera or the video processing device.

[0016] According to another aspect of this application, a video occlusion processing system is also provided, comprising: an identification module, configured to identify one or more target objects from a video frame, generate a corresponding layer for each target object, and record the position information of each target object in the video frame, wherein the position of each target object in its corresponding layer is the same as its position in the video frame; an acquisition module, configured to acquire the target object to be occluded from the one or more target objects, wherein the target object to be occluded is at least one; a determination module, configured to acquire the position information of the target object to be occluded in the video frame, and determine the portion occupied by the target object to be occluded in the video frame based on the layer corresponding to the target object to be occluded and the position information; and a processing module, configured to occlude the portion occupied by the target object to be occluded in the video frame.

[0017] According to another aspect of this application, a video processing apparatus is also provided, including a memory and a processor; wherein the memory is used to store one or more computer instructions, wherein the one or more computer instructions are executed by the processor to implement the above-described method steps.

[0018] According to another aspect of this application, a readable storage medium is also provided, on which computer instructions are stored, wherein the computer instructions, when executed by a processor, implement the above-described method steps.

[0019] In this embodiment, one or more target objects are identified from a video frame, and a corresponding layer is generated for each target object, wherein the position of each target object in its corresponding layer is the same as its position in the video frame; the target object to be occluded is obtained from the one or more target objects, wherein there is at least one target object to be occluded; the portion occupied by the target object to be occluded in the video frame is determined based on the position of the target object in its corresponding layer; and the portion occupied by the target object to be occluded in the video frame is occluded. This application solves the problem in the prior art where target occlusion occludes all targets of the same type, thus failing to occlude only some targets, and achieves partial target occlusion. Attached Figure Description

[0020] The accompanying drawings, which form part of this application, are used to provide a further understanding of this application. The illustrative embodiments and descriptions of this application are used to explain this application and do not constitute an undue limitation of this application. In the drawings:

[0021] Figure 1 This is a schematic diagram of a video system usage scenario according to an embodiment of this application. Figure 1 ;

[0022] Figure 2 This is a schematic diagram of a video system usage scenario according to an embodiment of this application. Figure 2 ;

[0023] Figure 3 This is a flowchart of a video occlusion processing method according to an embodiment of this application;

[0024] Figure 4 This is a schematic diagram of a data frame format according to an embodiment of this application;

[0025] Figure 5 This is a schematic diagram illustrating the occlusion effect according to an embodiment of this application;

[0026] Figure 6 This is a schematic diagram of the occlusion processing flow according to an embodiment of this application;

[0027] Figure 7 This is a schematic diagram of the modules involved in the layer segmentation process according to an embodiment of this application;

[0028] Figure 8 This is a schematic diagram of a target occlusion process according to an embodiment of this application;

[0029] Figure 9 This is a schematic diagram of target coordinates and target relationships according to embodiments of this application; and,

[0030] Figure 10 This is a schematic diagram of the interface for selecting the occlusion type according to an embodiment of this application. Detailed Implementation

[0031] It should be noted that, unless otherwise specified, the embodiments and features described in this application can be combined with each other. This application will now be described in detail with reference to the accompanying drawings and embodiments.

[0032] It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowchart, in some cases the steps shown or described may be executed in a different order than that shown here.

[0033] The following embodiments involve the processing of video images. The technical terms used in the following embodiments will be explained first.

[0034] Target detection

[0035] After a webcam captures video footage, the process of detecting objects (or simply targets) from the captured video is called object detection. The target object can be a living organism or part of a living organism, or it can be an object (e.g., a vehicle or its license plate). The target object can be moving or stationary. Deep learning algorithms can be used for object detection, enabling the identification of targets in the video.

[0036] XVR / DVR / NVR

[0037] In video systems, the terms NVR, DVR, and XVR are frequently used. Regardless of whether it's NVR, DVR, or XVR, they all refer to a type of hard disk recorder in a video system. In the following implementation, NVR, DVR, and XVR will be collectively referred to as hard disk recorders. NVR is an abbreviation for Network Video Recorder, also known as a network hard disk recorder.

[0038] The primary function of an NVR is to receive, store, and manage digital video streams transmitted from network cameras (IPCs), thereby leveraging the advantages of a distributed architecture brought about by networking. An NVR allows for the simultaneous viewing, browsing, playback, management, and storage of video data from multiple network cameras.

[0039] DVR stands for Digital Video Recorder. Unlike traditional analog video recorders, it uses hard disk recording, hence it's often called a hard disk recorder. A DVR is a computer system that performs image processing and storage, capable of recording and audio (images, audio, and moving frames) for extended periods.

[0040] XVR, or Digital Video Recorder, enables analog video and audio encoding, recording and video management, and analog-to-digital conversion, offering strong adaptability to various scenarios. Simply put, XVR combines the functions of NVR and DVR, allowing for seamless switching between them.

[0041] Figure 1 This is a schematic diagram of a video system usage scenario according to an embodiment of this application. Figure 1 ,like Figure 1As shown, a hard disk recorder (XVR / DVR / NVR) connects to a camera, such as a webcam or other recording device. The recorder acquires the video stream from the camera via a BNC cable or Ethernet cable. After acquiring the video stream, the recorder can either transmit it to a connected monitor for playback via an HDMI cable or a VGA cable, or upload the video stream to the cloud via a network. The uploaded video stream can be processed or unprocessed by the recorder. Once uploaded to the cloud, the mobile device can then access the desired bitrate from the cloud.

[0042] Figure 2 This is a schematic diagram of a video system usage scenario according to an embodiment of this application. Figure 2 ,like Figure 2 As shown, a network video recorder (NVR) connects to one or more network cameras (IPCs) via a network. Figure 1 Three webcams were used, and a network video recorder captured video feeds from them. The network video recorder was also connected to a computer, allowing the viewer to see the webcam feeds.

[0043] exist Figure 2 and Figure 1 In the illustrated scenario, target detection can be implemented using either a webcam or a hard disk recorder (DVR). The DVR can provide real-time preview of video footage, playback of recorded footage, reception and display of alarm images, and playback of alarm videos. Similarly, if a computer is connected, the computer can acquire video from the DVR or directly receive video from the webcam, enabling preview and playback functions as well. Figure 2 and Figure 1 In this scenario, it's necessary to perform targeted target occlusion on the footage viewed by the hard disk recorder and the computer, and then display the occluded image. It should be noted that the above... Figure 2 and Figure 3 The following two scenarios illustrate the need for target occlusion, but are not limited to them. The following implementation methods can be applied to any scenario where target occlusion is required.

[0044] When performing target occlusion, one can choose to uniformly occlude a certain type of target. For example, vehicles can be selected as targets. In this case, after identifying vehicles in the video, they can be uniformly occluded. This method is relatively easy to implement, but it cannot occlude some vehicles without occluding others. In the fields of video conferencing or live video streaming, there is another occlusion method. In this method, the outlines of participants in the video conference or the host conducting the live video stream can be identified, and then the background outside the outlines is blurred. This method can only blur the background in the video image and cannot partially occlude the target.

[0045] To address the aforementioned problems, a video occlusion processing method is provided in the following embodiments. Figure 3 This is a flowchart of a video occlusion processing method according to an embodiment of this application. Figure 1 The steps shown can be performed by any device with processing capabilities, for example, by the device described above. Figure 2 and Figure 3 The hard disk recorder shown in the diagram is used to perform this action, but other devices can also be used. In the following embodiments, a device capable of performing the following steps will be referred to as a video processing device. It should be noted that... Figure 3 The steps shown can also be performed by multiple devices. For example, a webcam can perform the step of identifying the target object in the video frame, and then a video processing device can perform the remaining steps, etc. Figure 3 The subject of the intermediate steps is not limited in the following embodiments; any device that performs the following steps shall be within the protection scope of the embodiments of this application. The following describes... Figure 3 The steps involved in the method described are explained.

[0046] Step S302: Identify one or more target objects (or simply targets) from the video frame, and generate a corresponding layer for each target object. The position of each target object in its corresponding layer is the same as its position in the video frame. This step involves video frames, which are composed of video frames. A video frame can be understood as a single frame within a video frame.

[0047] In this step, identifying one or more target objects from the video footage can be achieved in any way. For example, a deep learning-based neural network model can be used for identification. The neural network model can be trained using multiple pre-labeled training data sets, each containing an image and one or more labels to identify the target objects in the image. After training the neural network model using multiple sets of training data, the trained model can be used to identify the target objects. The video footage can then be input into the neural network model, which will output the target objects present in the video footage. Alternatively, other methods can be used to identify the target objects. The specific method of target object identification does not affect the solution to the technical problem in the following embodiments, and will not be elaborated upon here.

[0048] After the target object is identified, a corresponding layer is generated for the target object in this step. Using the corresponding layer for occlusion can achieve selective occlusion of the target object. This will be explained in the following implementation.

[0049] Step S304: Obtain the target object that needs to be occluded from the one or more target objects, wherein the target object that needs to be occluded is at least one.

[0050] In this step, a software interface can be provided to the user, displaying all the target objects identified from the video footage. The user can select the target objects to be occluded or not. If the user selects the unoccluded targets, the remaining targets are those that need to be occluded. When multiple types of target objects are identified from the video footage, such as bicycles and car license plates, the user can choose one type to occlude or not. For example, when identifying bicycles, the license plates need to be occluded. In this case, the user can directly select "license plates" as the target object type, and all license plates will be occluded. With this optional implementation, the user no longer needs to select each target object individually, but only needs to select one type. Alternatively, users can select whether to occlude or not based on the attributes of the target object. For example, if a user needs to identify a red car, after selecting the vehicle type of target object, the user can also select red. In this case, all other target objects except the red car will be occluded, which helps the user to quickly select the target object.

[0051] In another implementation, the user can upload an image through the software interface. The system can then identify target objects within the uploaded image, compare them with those in the video footage, and identify those with a similarity exceeding a threshold. These target objects are then configured to either require occlusion or not. For example, if a user needs to identify a vehicle, after uploading a photo of the vehicle, it is compared with target objects in the video footage. Target objects with a similarity exceeding the threshold are configured not to require occlusion, while other target objects are configured to require occlusion. There are many other ways to select target objects to be occluded or not, which will not be elaborated here. Users can flexibly choose according to the needs of the scenario in practical applications.

[0052] In another alternative implementation, the user can also input time information, such as a time period, and then occlude targets in video frames belonging to that time period. The selected targets can still be chosen using the methods described above. The user can also select different video channels; for example, the user can input a video channel number, and then occlude targets in the video frames corresponding to that channel number.

[0053] In this step, obtaining the target objects that need to be occluded from the one or more target objects may include at least one of the following: receiving time information and / or video channel number input by the user, and obtaining the target objects that need to be occluded from the video frames corresponding to the time information and / or video channel number; receiving a target object selected by the user from one or more target objects identified in the video frame, and using the target object selected by the user as the target object that needs to be occluded or does not need to be occluded; receiving the type or attribute of the target object input by the user, and using the target object of the type or the target object with the attribute as the target object that needs to be occluded or does not need to be occluded; receiving an image input by the user, identifying the target object from the image, and using the target object that is the same as the target object identified in the image from one or more target objects identified in the video frame as the target object that needs to be occluded or does not need to be occluded.

[0054] Step S306: Determine the portion of the video frame occupied by the target object to be occupied in the layer corresponding to the target object to be occupied.

[0055] In this step, since the target object needs to be occluded, the portion occupied by the target object in the video frame can be obtained through this step; this portion is the part that needs to be occluded. In step S302, all target objects identified from the video frame have been recorded, and a layer corresponding to each target object has been generated. In this step S306, the layer corresponding to the target object that needs to be occluded is found from the layers generated in step S302. Then, based on the position of the target object in that layer, the portion occupied by the target object in the video frame that needs to be occluded is determined.

[0056] Step S308: Obscure the portion of the target object that needs to be obscured in the video frame.

[0057] When occupying the portion of the target object in the video frame, various occupancy methods can be used. This step does not limit the specific occupancy method used; as long as the portion of the target object in the video frame is occupied, it will achieve the same effect.

[0058] In the above steps, after identifying the target object from the video image, a selection can be made from the identified target objects, and then a portion of the identified target objects in the video image can be occluded based on the selection result. Therefore, the above steps solve the problem in the prior art where target occlusion is performed on all targets of the same type, thus failing to achieve occlusion of some targets, and achieve occlusion of some targets.

[0059] There are many ways to generate layers. For example, each layer can use a first pixel value to identify the target object in that layer and a second pixel value to identify the rest of the layer, where the first pixel value and the second pixel value are different. In this optional embodiment, the layer uses two pixel values ​​to identify the target object. This approach can minimize the amount of data in the layer, saving storage space. On the other hand, because two pixel values ​​are used for differentiation, it is also easy to extract the target object from the layer, improving computational efficiency.

[0060] Ideally, the first and second pixel values ​​should be chosen as complementary colors to maximize the difference between them. In a proposed implementation, binarization can be used, where each target object's layer is a binarized layer. The first pixel value is either 1 or 0, and the second pixel value is either 1 or 0. Using a binarized layer, each pixel can be represented by one bit. For example, the pixel value corresponding to the target object is 1 (white), while the pixel values ​​of all other pixels in the layer are 0 (black). Binarization further reduces the storage space occupied by the layer and improves computational efficiency.

[0061] The purpose of layers is to indicate the outline of a target object in the video frame or the portion it occupies in the video frame. Therefore, the size and resolution of the layer corresponding to each target object can be the same as the video frame, making it easy to map the target object of the layer to the video frame. Of course, they can also be different. For example, if it is necessary to further save storage space, the size and / or resolution of the layer corresponding to the target object can be scaled down proportionally. It can simply be scaled back up proportionally during calculation. It should be noted that although this method saves storage space, it increases the amount of calculation and the accuracy of the result is not enough. This point needs to be taken into account in practical applications.

[0062] The added layer in the above embodiments can be saved as part of the video frame, that is, each video frame or each video keyframe can save the layer. This way, during video playback, occluded objects in the video will remain occluded. Existing video frame (also called data frame) formats do not have fields for saving layers. Therefore, in an optional embodiment, a new format of video frame is provided, which may include a first part and a second part. These two parts are described below:

[0063] The first part is used to save the content of the original video frame corresponding to the video image;

[0064] The second part stores the following data or the second part stores index information, wherein the index information is used to indicate the storage location of the following data, the data including: a part for storing the resolution of the video frame and the resolution of the layer, a part for storing the number of target objects detected in the video frame, and indication information for indicating the last target object stored in the video frame, and a part for storing the layer corresponding to the target object, wherein there are N parts for storing the layer corresponding to the target object, and each target object corresponds to one part for storing the layer corresponding to the target object, where N is the number of target objects detected in the video frame.

[0065] It should be noted that, in Figure 4 The illustrated steps determine the portion to be occluded based on the area occupied by the target object in the video frame. Coordinate information can be introduced when determining the occluded portion. This coordinate information can be located in the layer and / or in the video frame. The coordinate information includes the coordinates of points on the border surrounding the target object, such as the coordinates of the vertices of the polygon surrounding the target object. Coordinate information can be used for auxiliary positioning. For example, a polygon can be used to surround the target object, and the coordinates of several vertices of that polygon can be saved. This method is relatively easy to implement, but the resulting target object outline is relatively simple. This polygon can be used for auxiliary positioning. In another embodiment, the coordinate information of the polygon vertices and the outline information can be combined. The outline information can be represented using the target object in the corresponding layer (therefore, in the following embodiments, the layer can also be called the outline layer). Using the outline layer, the target object can be accurately occluded, improving the accuracy of the occlusion. Therefore, as an optional implementation, the second part can also store the coordinate information of the target object in the layer or the video frame. This coordinate information includes the coordinates of points on the border surrounding the target object. The border can be located using the coordinates of these points, and the target object can then be initially located using this border. There are various ways to select the points on the border. For example, all points on the border can be selected; another example is selecting a point at predetermined intervals of pixels; yet another example is selecting points at the turning points on the border; and yet another example is selecting the vertices of a polygon as points on the border, etc. Regardless of how the points on the border are selected, as long as the selected points can be used to depict the outline of the border, it is acceptable.

[0066] There are two ways to save the layer corresponding to the target object: one is to save the pixel value of each pixel in the layer, and the other is to save the layer as an image file. These two methods can be used alone or in combination.

[0067] The following example illustrates the format of a data frame. Figure 4 This is a schematic diagram of the data frame format according to an embodiment of this application, such as... Figure 4As shown, the data frame format includes: the original video frame, a private information field (used to store the resolution of the video frame and the number of identified targets), and a section for storing the location information of the targets. Each section storing the target location information can include two fields: one field stores the sequence number, the identification information (ID number) of target 1, and the coordinate information of target 1; the other field identifies the layer information of target 1. Video frames saved using this data frame format directly carry the location information of all targets. Therefore, each time a video composed of video frames in this format is played, the target to be occluded can be selected. For example, during video playback, target 1 can be selected to be occluded; after downloading the video to another playback device, target 2 can be selected to be occluded during subsequent playback. That is, different targets can be selected to be occluded each time the video is played. It should be noted that the resolution of the video frame and the resolution of the layer can be the same. If they are the same, then... Figure 4 As shown, one field can be used to store the resolution (the resolution of a video frame is stored in the image, which is also the resolution of the layer). If they are different, two fields can be used to store the resolution of the video frame and the resolution of the layer respectively.

[0068] In another alternative implementation, in Figure 5 The data frame format shown can also include a fourth part, which can store the default occlusion target. For example, for a video A that needs to be distributed to users 1, 2, and 3, the fourth part of the video A distributed to user 1 can be set to the default occlusion target 1, the video distributed to user 2 can be set to the default occlusion target 2, and the video distributed to user 3 can be set to the default occlusion target 3. In this way, although a single video is used, the occluded target in the video distributed to different users is different, providing flexibility in occlusion. A field can also be added to the data frame format to indicate whether the video allows users to select the occluded target. If allowed, occlusion can be performed according to the user's selection; otherwise, the default occlusion configuration is used.

[0069] After using video frames with layers, in an optional implementation, determining the portion of the target object to be occupied in the video frame by identifying the position of the target object in the layer corresponding to the target object to be occupied can include the following steps: based on the resolution of the layer corresponding to the target object to be occupied and the resolution of the video frame, mapping the area containing pixels with the first pixel value in the layer corresponding to the target object to be occupied onto the video frame, thus obtaining the portion occupied by the target object to be occupied in the video frame. If coordinate information is saved, the coordinate information of the target object can also be obtained, and the range of the target object in the video frame can be located based on the coordinate information; then, within the range, the area containing pixels with the first pixel value in the layer corresponding to the target object to be occupied can be mapped onto the video frame. Through this optional implementation, the coordinate information and layers can be fully utilized by using layers representing the portion of the target object to be occupied in the video frame, making the acquisition of the portion to be occupied more efficient.

[0070] When occluding objects, different occlusion methods can be used. The following implementation provides three optional occlusion methods, which are described below.

[0071] Method 1

[0072] The portion of the video frame occupied by the object to be obscured is filled with one or more colors. In this method, one color can be used to fill the obscured portion, such as red or yellow, or two or more colors can be used, for example, half of the obscured portion can be filled with red and the other half with yellow. This method can also be called color filling.

[0073] Method 2

[0074] The target object to be occluded occupies a portion of the video frame as multiple pixel blocks, each of which is filled with a single color. In this method, each pixel block can include multiple pixels, and the color of each pixel block can be randomly selected; alternatively, the occupied portion can be divided into multiple pixel blocks, and for each pixel block, the average pixel value of all pixels in that block is obtained and used to fill the pixel block. This method can also be called pixel block occlusion, mosaic occlusion, or mosaic masking, etc.

[0075] Method 3

[0076] The process involves obtaining a pre-saved background image corresponding to the video frame, extracting the background portion corresponding to the target object from the background image based on its position information within the background image, and then filling the extracted background portion into the corresponding portion of the video frame. The background image and the video frame are both images taken from the same location at the same angle, and the background image does not contain the target object. In this method, the transparency of the background portion can also be set; that is, the extracted background portion is given a predetermined transparency, and the background portion with the set transparency is then filled into the corresponding portion of the video frame. This method can be called background occlusion or transparent occlusion.

[0077] Figure 5 This is a schematic diagram illustrating the occlusion effect according to an embodiment of this application. Figure 5 The image shows the occlusion effects of the three methods mentioned above, such as... Figure 5 As shown, the initial image a (i.e., the unoccluded video frame) shows multiple original targets, including but not limited to living organisms and license plates. Images b-d show the effects under different occlusion requests. Image b shows the effect of full target occlusion, where all targets are occluded along their outlines, including near and far targets. Image c shows the effect of partial or specified target occlusion, where the target object requested for protection is occluded along its outline, while the remaining targets display the original image. Image d shows the unoccluded effect, where the entire image displays the original content. Images e-g show three target occlusion display effects. Image e indicates occlusion with a specified target color (indicated by stripes in this image), and the color can be selected as needed, for example... Figure 5 The entire outline of the target can be displayed as pure yellow. Image f represents target pixel color block occlusion, also known as mosaic occlusion or mosaic masking, where the target area is displayed as several pixel color blocks. Image g represents target background occlusion or transparency occlusion, where the target outline area is displayed as a semi-transparent display of the original background image (in... Figure 6 (Not shown in the image).

[0078] The above embodiments can be applied to various video processing systems. For example, they can be applied to a video system that may include: a camera for capturing video images; and a video processing device connected to the camera for receiving the video images and executing the method steps in the optional embodiments described above. It should be noted that the step of identifying one or more target objects from the video images is performed by the camera or the video processing device. The video captured by the camera is generally a video stream, and the video images are obtained from the video frames extracted from the stream. When performing occlusion, since one of the three methods described above can be selected for occlusion, and background occlusion requires the use of a background image, the obtained background image (or background information) can be imported first. Figure 6 This is a schematic diagram of the occlusion processing flow according to an embodiment of this application. Figure 6 In this process, each step can be implemented through a software module, such as... Figure 7 As shown, it can include modules such as: importing background information, object detection, object segmentation and generating layer recording data, determining occlusion type, object occlusion, and video processing. The following sections will explain each of these modules.

[0079] Import background information

[0080] This module is used to obtain the blank background image of the camera (i.e., IPC) scene in advance and import it into the NVR for subsequent background filling when the target is occluded.

[0081] Target detection

[0082] Targets include, but are not limited to, living organisms and license plates. This module can use general deep learning algorithms to detect targets in the video, ensuring that targets appearing in the video are detected and recorded in a timely manner.

[0083] Generate layers and target segmentation

[0084] This module can also include four sub-modules. Figure 7 This is a schematic diagram of the modules involved in the layer segmentation process according to an embodiment of this application, such as... Figure 1 As shown, it includes: object detection, object segmentation, layer generation, and video storage modules. Object detection obtains the target and its coordinate information. For each target, segmentation or contour finding is performed. Then, a separate layer is generated for each target, with the layer size consistent with the original image (i.e., the video frame), and the target's location is consistent with the original image. Figure 4 The layer is a binarized result; pixels within the target outline have a value of 1, while pixels in other areas have a value of 0. The frame format used for storage is as follows: Figure 10As shown, the data frame format includes: the original video frame, a private information field (used to store the resolution of the video frame and the number of identified targets), and a section for storing the location information of the targets. Each section storing the target location information can include two fields: one field stores the sequence number, the ID number of target 1, and the coordinate information of target 1; the other field identifies the layer information of target 1. The target's layer information includes two forms: form one is a data set of the pixel values ​​of each pixel in the layer, and form two is a target layer image.

[0085] Determine the type of occlusion

[0086] An occlusion request is initiated by a device or an external computer. The occlusion request carries the occlusion type, which can include three types: full target occlusion, partial target occlusion, and no occlusion.

[0087] Figure 10 This is a schematic diagram of the interface for selecting the occlusion type according to an embodiment of this application, such as... Figure 8 As shown, the occlusion interface supports three options: full occlusion, full display, and partial occlusion. This interface includes, but is not limited to, the NVR configuration interface, the PC software interface, and the client interface.

[0088] In partial occlusion mode, first determine the video range to be occluded by selecting the channel number and start and end time points. All targets within the displayed video clip range are shown as small images. These small images are extracted using coordinate information or obtained based on the mapping result between the layer and the original video frame. Only one small image is displayed for each target ID. When the user selects a small image that they deem necessary to occlude, the corresponding area of ​​the video frame containing the selected target with the same ID will be occluded.

[0089] Target occlusion and video processing

[0090] Figure 8 This is a schematic diagram of the target occlusion process according to an embodiment of this application, such as... Figure 9 As shown, the process includes three steps: specifying the target, determining the target location, and filling the target area with pixels. These three steps can be implemented through three sub-modules.

[0091] Figure 9 This is a schematic diagram of target coordinates and target relationships according to the embodiments of this application, such as... Figure 1 As shown, after specifying the target to be occluded, the system obtains the target's layer, its coordinates, and the original image resolution. The area with a pixel value of 1 (the target region) in the target layer is mapped onto the original image, allowing the target's outline to be obtained. This outline is then filled accordingly. Using the target's coordinates (4-point coordinates) within its layer improves the positioning speed.

[0092] Target occlusion supports three modes: target specified color occlusion, target mosaic color block occlusion, and target background occlusion. If the target specified color occlusion mode is selected, the RGB values ​​of the target area in the original image correspond to the specified color values. If the target mosaic color block occlusion mode is selected, the target outline is divided into several n*n pixel color blocks, and each block displays the average color value of all pixels in that area. If the target background occlusion mode is selected, a pre-saved background image is obtained, and the target position of the target layer is used to cut out the image from the background image, then the cut-out image is added to the original image to make the target transparent.

[0093] In the occlusion scheme described in the above optional implementation, the device performing the occlusion process (such as a hard disk recorder, computer, or any device with processing capabilities) pre-records background image information. Then, it identifies targets in the original image to obtain one or more target objects. The outline of each target object is segmented and recorded as a single layer, and the corresponding coordinate position of the target is recorded in the private information within the frame information. When a video request is received, the corresponding target is occluded or not occluded as needed. Three options are available for the occluded target effect: solid color background, mosaic pixel blocks, and a semi-transparent background effect, thereby achieving the purpose of information protection.

[0094] In this embodiment, an electronic device is provided, including a memory and a processor. The memory stores a computer program, and the processor is configured to run the computer program to perform the methods described in the above embodiments.

[0095] The aforementioned program can run on a processor or be stored in memory (or a computer-readable medium). Computer-readable media include both permanent and non-permanent, removable and non-removable media, and information storage can be achieved by any method or technology. Information can be computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transfer medium that can be used to store information accessible by a computing device.

[0096] These computer programs may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes ​ The steps for the functions specified in one or more boxes can be implemented using different modules, and different steps can be implemented using different modules.

[0097] This embodiment provides such a system, referred to as a video occlusion processing system, comprising: a recognition module, configured to recognize one or more target objects from a video frame and generate a corresponding layer for each target object, wherein the position of each target object in its corresponding layer is the same as its position in the video frame; an acquisition module, configured to acquire the target object to be occluded from the one or more target objects, wherein the target object to be occluded is at least one; a determination module, configured to determine the portion occupied by the target object to be occluded in the video frame based on the position of the target object in the layer corresponding to the target object to be occluded; and a processing module, configured to occlude the portion occupied by the target object to be occluded in the video frame.

[0098] The system or apparatus is used to implement the functions of the methods in the above embodiments. Each module in the system or apparatus corresponds to each step in the method, as has been described in the method and will not be repeated here.

[0099] Optionally, in each layer, a first pixel value is used to identify the target object in that layer, and a second pixel value is used to identify the rest of the layer other than the target object, wherein the first pixel value and the second pixel value are different.

[0100] Optionally, the video frame and the layer are saved using a video frame. The video frame includes a first part and a second part, wherein: the first part is used to save the content of the original video frame corresponding to the video frame; the second part saves the following data or index information, wherein the index information is used to indicate the saving location of the following data, the data including: a part for saving the resolution of the video frame and the resolution of the layer, a part for saving the number of target objects detected in the video frame, and at least one of the following: indication information for indicating the last target object saved in the video frame, and a part for saving the layer corresponding to the target object, wherein there are N parts for saving the layer corresponding to the target object, and each target object corresponds to one part for saving the layer corresponding to the target object, where N is the number of target objects detected in the video frame.

[0101] Optionally, saving the layer corresponding to the target object includes: saving the pixel value of each pixel in the layer, or saving the layer as an image file; and / or, the second part further saves the coordinate information of the target object in the layer and / or in the video frame, wherein the coordinate information includes the coordinates of points on the border surrounding the target object.

[0102] Optionally, the determining module is used to: map the area containing pixels with the first pixel value in the layer corresponding to the target object to be occluded to the video frame according to the resolution of the layer corresponding to the target object to be occluded and the resolution of the video frame, so as to obtain the part occupied by the target object to be occluded in the video frame.

[0103] Optionally, the determining module is used to obtain the coordinate information of the target object and locate the range of the target in the video frame according to the coordinate information; wherein, the coordinate information includes the coordinates of points on the border surrounding the target object; within the range, the area where the pixel value of the pixel in the layer corresponding to the target object to be occluded is the first pixel value is mapped onto the video frame.

[0104] Optionally, the acquisition module is used to acquire the target object to be occluded by at least one of the following:

[0105] Receive time information and / or video channel number input by the user, and obtain the target object to be occluded from the video frame corresponding to the time information and / or the video channel number;

[0106] The system receives a user's selection of one or more target objects from the video frame, and then uses the user-selected target object as either a target object that needs to be occluded or a target object that does not need to be occluded.

[0107] Receive the type or attribute of the target object input by the user, and designate the target object of the type or the target object with the attribute as the target object that needs to be occluded or the target object that does not need to be occluded;

[0108] The system receives an image input by the user, identifies target objects from the image, and selects target objects from one or more target objects identified from the video frame that are the same as the target objects identified in the image as target objects that need to be occluded or do not need to be occluded.

[0109] Optionally, the processing module is configured to: fill the portion of the video frame occupied by the target object to be occupied with one or more colors; and / or, divide the portion of the video frame occupied by the target object to be occupied into multiple pixel blocks, wherein each pixel block is filled with one color; and / or, acquire a pre-saved background image corresponding to the video frame, extract the background portion corresponding to the target object from the background image according to the position information of the target object to be occupied in the background image, and fill the extracted background portion into the portion of the video frame; wherein the background image and the video frame are images taken from the same position at the same angle, and the background image does not contain the target object.

[0110] Optionally, the processing module is configured to: divide the occupied portion into the plurality of pixel blocks, for each of the plurality of pixel blocks, obtain the average pixel value of all pixels in the pixel block, and fill the pixel block with the average pixel value; and / or, the processing module is configured to: set a predetermined transparency for the extracted background portion, and fill the background portion with the set transparency into the portion of the video frame.

[0111] In the above optional embodiments, target occlusion can be achieved for specified targets. The target occlusion scheme can be applied to multiple scenarios, such as video preview, playback, and download, where specified targets can be selectively occluded. It should be noted that different targets can be specified for occlusion during video preview, playback, and download. In specific implementation, the above optional embodiments use a contour layer for each target to obtain its position information. This position information is used to determine which targets can be displayed and which need to be occluded. A new data frame format is also provided for target occlusion. The above optional embodiments also provide three occlusion modes: target-specified color occlusion, target mosaic color block occlusion, and target background occlusion. These three occlusion modes can be flexibly selected as needed during implementation. Therefore, the above optional embodiments solve the problem in the prior art where target occlusion applies to all targets of the same type, thus failing to occlude only some targets, and achieve partial target occlusion.

[0112] The above are merely embodiments of this application and are not intended to limit the scope of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of the claims of this application.

Claims

1. A video occlusion processing method, characterized in that, include: One or more target objects are identified from the video frame, a corresponding layer is generated for each target object, and the video frame and the layer are saved using video frames, wherein the position of each target object in its corresponding layer is the same as the position of the target object in the video frame; The target object that needs to be occluded is selected from the one or more target objects, wherein the target object that needs to be occluded is at least one; The portion of the video frame occupied by the target object to be occupied is determined based on the position of the target object in the layer corresponding to the target object to be occupied. The portion of the target object that needs to be occupied in the video frame is occupied.

2. The method according to claim 1, characterized in that, Each layer uses a first pixel value to identify the target object in that layer, and a second pixel value to identify the rest of the layer other than the target object. The first pixel value and the second pixel value are different.

3. The method according to claim 1, characterized in that, The video frame comprises a first part and a second part, wherein: The first part is used to save the content of the original video frame corresponding to the video image; The second part stores the following data or the second part stores index information, wherein the index information is used to indicate the storage location of the following data, the data including: a part for storing the resolution of the video frame and the resolution of the layer, a part for storing the number of target objects detected in the video frame, and an indication information for indicating the last target object stored in the video frame, and a part for storing the layer corresponding to the target object, wherein there are N parts for storing the layer corresponding to the target object, and each target object corresponds to one part for storing the layer corresponding to the target object, where N is the number of target objects detected in the video frame.

4. The method according to claim 3, characterized in that, Saving the layer corresponding to the target object includes: saving the pixel value of each pixel in the layer, or saving the layer as an image file; And / or, The second part also stores the coordinate information of the target object in the layer and / or in the video frame, wherein the coordinate information includes the coordinates of points on the border surrounding the target object.

5. The method according to claim 2, characterized in that, Determining the portion of the video frame occupied by the target object to be occupied in the video frame based on the position of the target object in the layer corresponding to the target object to be occupied includes: Based on the resolution of the layer corresponding to the target object to be occluded and the resolution of the video frame, the area containing pixels with the first pixel value in the layer corresponding to the target object to be occluded is mapped onto the video frame to obtain the portion occupied by the target object to be occluded in the video frame.

6. The method according to claim 5, characterized in that, Mapping the area containing pixels with the first pixel value in the layer corresponding to the target object to be occluded onto the video frame includes: The coordinate information of the target object is obtained, and the range of the target object in the video frame is located based on the coordinate information; wherein, the coordinate information includes the coordinates of points on the border surrounding the target object; Within the specified range, the area containing pixels with the first pixel value in the layer corresponding to the target object to be occluded is mapped onto the video frame.

7. The method according to any one of claims 1 to 4, characterized in that, The target objects that need to be occluded from the one or more target objects include at least one of the following: Receive time information and / or video channel number input by the user, and obtain the target object to be occluded from the video frame corresponding to the time information and / or the video channel number; The system receives a user's selection of one or more target objects from the video frame, and then uses the user-selected target object as either a target object that needs to be occluded or a target object that does not need to be occluded. Receive the type or attribute of the target object input by the user, and designate the target object of the type or the target object with the attribute as the target object that needs to be occluded or the target object that does not need to be occluded; The system receives an image input by the user, identifies target objects from the image, and selects target objects from one or more target objects identified from the video frame that are the same as the target objects identified in the image as target objects that need to be occluded or do not need to be occluded.

8. The method according to any one of claims 1 to 4, characterized in that, Occluding the portion of the video frame occupied by the target object that needs to be occupied includes: Fill the portion of the video frame occupied by the target object that needs to be obscured with one or more colors; and / or, The portion of the target object to be occluded in the video frame is divided into multiple pixel blocks, wherein each pixel block is filled with a single color; and / or, Obtain the background image corresponding to the pre-saved video frame; extract the background portion corresponding to the target object from the background image according to the position information of the target object to be occluded in the background image; and fill the extracted background portion into the portion of the video frame; wherein the background image and the video frame are images taken from the same position and at the same angle, and there is no target object in the background image.

9. The method according to claim 8, characterized in that, Dividing the portion occupied by the target object to be occupied in the video frame into multiple pixel blocks includes: dividing the occupied portion into the multiple pixel blocks; for each pixel block, obtaining the average pixel value of all pixels in the pixel block; and filling the pixel block with the average pixel value. And / or, Filling the extracted background portion into the portion of the video frame includes: setting a predetermined transparency for the extracted background portion, and filling the background portion with the set transparency into the portion of the video frame.

10. A video system, characterized in that, include: A camera is used to capture video footage. A video processing device, connected to the camera, is used to receive the video frame and execute the method according to any one of claims 1 to 9, wherein the step of identifying one or more target objects from the video frame according to any one of claims 1 to 9 is executed by the camera or the video processing device.

11. A video occlusion processing system, characterized in that, include: The recognition module is used to identify one or more target objects from the video frame, generate a corresponding layer for each target object, and save the video frame and the layer using video frames, wherein the position of each target object in its corresponding layer is the same as the position of the target object in the video frame; The acquisition module is used to acquire the target object that needs to be occluded among the one or more target objects, wherein the target object that needs to be occluded is at least one; The determining module is used to determine the portion of the target object to be occupied in the video frame based on the position of the target object in the layer corresponding to the target object to be occupied; The processing module is used to occupies the portion of the video frame occupied by the target object that needs to be occupied.

12. A video processing device, comprising a memory and a processor; wherein, The memory is used to store one or more computer instructions, wherein the one or more computer instructions are executed by the processor to implement the steps of the method according to any one of claims 1 to 9.

13. A readable storage medium having computer instructions stored thereon, wherein, When executed by a processor, the computer instructions implement the steps of the method described in any one of claims 1 to 9.

Images