Video data coding method, video data decoding method, readable medium and electronic equipment

Abstract

The invention relates to the technical field of image processing, and discloses a video data coding method and a video data decoding method, in the method, an image to be processed is divided into a plurality of image areas, and the image areas comprise a privacy area and a non-privacy area of the image to be processed. At a coding end, each image area can be coded to form coded data of an image to be processed, and the coded data is transmitted to a decoding end, namely, the coded data is transmitted in a single-frame image data packet. And the decoding end decodes the data sub-packets of each image area in the received image data packet to obtain a reconstructed image. Therefore, only one image is subjected to single-layer coding, and each data region (including the privacy region and the non-privacy region) of the image can be coded and decoded only once, so that the calculation overhead of the coding process and the decoding process can be reduced.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to a video data encoding method, decoding method, readable medium and electronic device. Background Art

[0002] Video surveillance technology is widely used in monitoring scenarios. For example, video surveillance technology is used to implement remote video surveillance of target areas to assist in public security management.

[0003] However, the video data collected by the image acquisition device may carry the user's private information (such as face, ID information, license plate number, etc.). If the video data containing the user's private information is leaked, it may cause losses to the user. Therefore, the demand for video privacy protection technology is getting higher and higher. Video privacy protection is to make the image in the private area unrecognizable by coding, blocking or scrambling the private area in the video data, while retaining the original image content in the non-private area.

[0004] At present, the encoding end usually inputs the privacy area image and the non-privacy area image of the video data into two encoders for encoding, and obtains a code stream (or bit stream, bit stream, bit stream) of the privacy area image and a code stream of the non-privacy area image and transmits them to the decoding end or storage. During decoding, the code stream of the privacy area image and the code stream of the non-privacy area image are also input into two decoders for decoding, and after obtaining the reconstructed privacy image and the reconstructed non-privacy image, the two images are fused.

[0005] It can be understood that the privacy area image and the non-privacy area image are encoded and decoded independently, and the privacy area image and the non-privacy area image are encoded and decoded respectively, that is, the non-privacy area and the privacy area of ​​the original image are encoded and decoded twice, which increases the computational overhead of the encoding process and the decoding process. Summary of the invention

[0006] The first aspect of the present application provides a video data encoding method, including: obtaining a first image in a video to be processed; dividing the first image into multiple image areas, wherein the multiple image areas include a first type of image area and a second type of image area, wherein the image area is an encoding unit; encoding the multiple image areas respectively to form first encoded image data, wherein the first encoded image data includes first area encoding data of the first type of image area and second area encoding data of the second type of image area; and sending the first encoded image data.

[0007] In an embodiment of the present application, the first image may be an image to be processed, the first type of image area may be a privacy area, the second type of image area may be a non-privacy area, the first area encoding data of the first type of image area may be a privacy code stream, and the second area encoding data of the second type of image area may be a non-privacy code stream.

[0008] The video data encoding method provided by the present application only performs single-layer encoding on an image, and generates a corresponding code stream (the transmission rate of each frame image data packet transmitted by the same code stream is the same), which can reduce the network bandwidth resources in the transmission. In addition, each data area of ​​the image (including the privacy area and the non-privacy area) is only encoded once, which can reduce the computational overhead of the encoding process.

[0009] In a possible implementation of the first aspect above, the first type of image area is a privacy area, and the second type of image area is a non-privacy area.

[0010] In a possible implementation of the first aspect above, the first category image area is encoded in the following manner to obtain first region encoding data: the first category image area of ​​the first image is masked to obtain a first category masked area corresponding to the first category image area; the first category image area and the first category masked area are encoded respectively to obtain first region encoding data, wherein the first region encoding data includes first category image area encoding sub-data and first category masked area encoding sub-data.

[0011] In the embodiment of the present application, the first region coded data may be the coded sub-data of the privacy region, the first type of masked region may be the privacy masked region, the first type of image region coded sub-data may be the privacy code stream, and the first type of masked region coded sub-data may be the privacy masked content code stream. The privacy region of the image to be processed is masked to obtain a privacy masked image, and the privacy masked content in the privacy masked image is encoded to obtain a privacy masked content code stream; the privacy region of the image to be processed is encoded to obtain a privacy code stream; the coded sub-data of the privacy region includes a privacy code stream and a privacy masked content code stream.

[0012] In a possible implementation of the first aspect above, the masking processing includes at least one of the following: mosaic filling, Gaussian blur, artificial intelligence content replacement, and overlaying content set by the user.

[0013] In a possible implementation of the first aspect, sending the first encoded image data includes: sending the first encoded image data through a first channel.

[0014] The second aspect of the present application provides a video data decoding method, including: receiving first encoded image data of a first image in a video to be processed, wherein the first encoded image data includes first area encoding data of a first type of image area of ​​the first image and second area encoding data of a second type of image area, and the image area is a coding unit; determining a privacy protection mode of the first encoded image data, wherein the privacy protection mode includes a first mode and a second mode; decoding the first encoded image data based on the privacy protection mode to obtain first decoded image data.

[0015] In the embodiment of the present application, the first decoded image data may be reconstruction data (first reconstruction data or second reconstruction data).

[0016] In a possible implementation of the second aspect above, the first category of image areas is a privacy area, and the second category of images is a non-privacy area; and the first area encoding data includes first category image area encoding sub-data and first category masked area encoding sub-data, wherein the first category masked area encoding sub-data is obtained by encoding the first category image area after masking processing.

[0017] In a possible implementation of the second aspect above, the privacy protection mode is the first mode, and the first encoded image data is decoded based on the privacy protection mode to obtain first decoded image data, including: having a first decoding authority corresponding to the first encoded image data, decoding the first type of image area encoded sub-data, and obtaining first area decoded data corresponding to the first area encoded data; having a second decoding authority corresponding to the first encoded image data, setting the image area corresponding to the first area encoded data in the first image to a preset pixel value, and obtaining first area decoded data corresponding to the first area encoded data, wherein the first decoding authority is higher than the second decoding authority.

[0018] In an embodiment of the present application, the first decoding authority may be a high user authority, the second decoding authority may be a low user authority, and the first area decoded data may be decoded data of a privacy area. When the privacy protection mode is the first mode: corresponding to the high user authority, the coding unit in the privacy area is decoded based on the privacy code stream to obtain the decoded data of the privacy area of ​​the coded sub-data corresponding to the privacy area. Corresponding to the low user authority, a preset pixel value (preset pixel value) is used for pixel filling to obtain the decoded data of the privacy area of ​​the coded sub-data corresponding to the privacy area.

[0019] In a possible implementation of the second aspect above, the privacy protection mode is the second mode, and the first encoded image data is decoded based on the privacy protection mode to obtain first decoded image data, including: having a first decoding authority corresponding to the first encoded image data, decoding the first type of image area encoded sub-data to obtain first area decoded data corresponding to the first area encoded data; having a second decoding authority corresponding to the first encoded image data, decoding the first type of masked area encoded sub-data to obtain first area decoded data corresponding to the first area encoded data, wherein the first decoding authority is higher than the second decoding authority.

[0020] In the embodiment of the present application, when the privacy protection mode is the second mode: corresponding to high user authority, the coding unit in the privacy area is decoded based on the privacy code stream to obtain the decoded data of the privacy area of ​​the coded sub-data corresponding to the privacy area. Corresponding to low user authority, the coding unit is decoded based on the privacy masking content code stream to obtain the decoded data of the privacy area of ​​the coded sub-data corresponding to the privacy area.

[0021] In a possible implementation of the second aspect above, the privacy protection mode is the first mode or the second mode, and the first encoded image data is decoded based on the privacy protection mode to obtain first decoded image data, including: decoding the second area encoded data to obtain second area decoded data corresponding to the second area encoded data.

[0022] In an embodiment of the present application, the second area decoded data may be decoded data of the non-privacy area. When the privacy protection mode is the first mode or the second mode: the coding unit is decoded based on the non-privacy code stream to obtain decoded data of the non-privacy area corresponding to the coded sub-data of the non-privacy area.

[0023] In a possible implementation of the second aspect, the method further includes: receiving region division information of the first encoded image data; and determining region encoding sub-data of each image region in the first image based on the region division information.

[0024] In the embodiment of the present application, the region division information may be coding unit division information, and the region coding sub-data may be coding unit sub-data.

[0025] A third aspect of the present application provides a readable medium on which instructions are stored. When the instructions are executed on an electronic device, the electronic device executes any one of the methods in the first aspect or the second aspect.

[0026] The fourth aspect of the present application provides an electronic device, comprising a memory for storing instructions executed by one or more processors of the electronic device, and a processor, which is one of the processors of the electronic device, for executing any one of the methods in the first aspect or the second aspect above.

[0027] A fifth aspect of the present application provides a program product, which includes instructions. When the instructions are executed by an electronic device, the electronic device can implement any one of the methods in the first aspect or the second aspect.

[0028] The sixth aspect of the present application provides a chip, characterized in that the chip system includes a processing circuit and a storage medium, and the storage medium stores computer program code; when the computer program code is executed by the processing circuit, any one of the methods in the first aspect or the second aspect is implemented. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] In order to more clearly illustrate the specific implementation methods of the present invention or the technical solutions in the prior art, the drawings required for use in the specific implementation methods or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some implementation methods of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0030] Figure 1 According to an embodiment of the present application, a schematic diagram of an application scenario is shown;

[0031] Figure 2 According to an embodiment of the present application, a schematic diagram of a compression framework is provided;

[0032] Figure 3 According to an embodiment of the present application, a schematic diagram showing a privacy image and a non-privacy image is provided;

[0033] Figure 4 According to an embodiment of the present application, a schematic flow diagram of a dual-stream privacy protection method is shown;

[0034] Figure 5a According to an embodiment of the present application, a flowchart of a video data encoding method and a video data decoding method is shown;

[0035] Figure 5b According to an embodiment of the present application, a schematic diagram of an implementation flow of a video data encoding method is shown;

[0036] Figure 6a According to an embodiment of the present application, a schematic diagram of a flow chart of an encoding process is shown;

[0037] Figure 6b According to an embodiment of the present application, a schematic block diagram of an encoding process is shown;

[0038] Figure 7 According to an embodiment of the present application, a flowchart of another encoding process is shown;

[0039] Figure 8 According to an embodiment of the present application, a flowchart of a decoding process is shown;

[0040] Figure 9a According to an embodiment of the present application, a schematic diagram of an implementation process of a low-authority decoding strategy is shown;

[0041] Figure 9b A schematic diagram of a data flow diagram of a decoding process is shown according to an embodiment of the present application;

[0042] Fig.10a According to an embodiment of the present application, a schematic diagram of an implementation process of a high-authority decoding strategy is shown;

[0043] Fig.10b A schematic diagram of a data flow diagram of a decoding process is shown according to an embodiment of the present application;

[0044] Fig.11 According to an embodiment of the present application, another flowchart of a decoding process is shown;

[0045] Fig.12a According to an embodiment of the present application, another implementation flow diagram of a low-authority decoding strategy is shown;

[0046] Figure 12b A schematic diagram of a data flow diagram of a decoding process is shown according to an embodiment of the present application;

[0047] Fig.13a According to an embodiment of the present application, a schematic diagram of an implementation process of a high-authority decoding strategy is shown;

[0048] Fig.13b A schematic diagram of a data flow diagram of a decoding process is shown according to an embodiment of the present application;

[0049] Fig.14 According to an embodiment of the present application, a structural schematic diagram of an electronic device is shown. DETAILED DESCRIPTION

[0050] The illustrative embodiments of the present application include, but are not limited to, a video data encoding method, a decoding method, a readable medium, and an electronic device.

[0051] In order to make the objectives, technical solutions and advantages of the present application clearer, the technical solutions in the embodiments of the present application will be clearly and comprehensively described below in conjunction with the accompanying drawings.

[0052] The video data encoding method and the video data decoding method provided in the embodiments of the present application can be applied to a client, and the client may include an image acquisition device and a terminal device, and may also include an image acquisition application and an image viewing application. The image acquisition device includes but is not limited to a camera, and electronic devices including a camera such as a mobile phone, a smart screen, a tablet computer, and a laptop computer. The terminal device includes but is not limited to a mobile phone, a smart screen, a tablet computer, a laptop computer, a netbook, a personal digital assistant (PDA), and other electronic devices with a display screen. Exemplary embodiments of electronic devices include but are not limited to devices equipped with Or electronic devices with other operating systems. Image acquisition applications include but are not limited to camera applications, conference applications, and other applications with image acquisition functions. Image viewing applications include but are not limited to photo album applications, video playback applications, and other applications with image viewing functions. The embodiments of this application do not impose any restrictions on the specific types of electronic devices and applications.

[0053] It can be understood that the image acquisition device and the terminal device can be two electronic devices or the same electronic device; the image acquisition application and the image viewing application can be two application programs or the same application program, and this application does not impose any specific restrictions on this.

[0054] Figure 1 According to an embodiment of the present application, a schematic diagram of an application scenario is shown. Figure 1 As shown, the image acquisition device 1, the image acquisition device 2, the image acquisition device 3, ... acquires video data, and then transmits the acquired video data to the terminal device 1, the terminal device 2, the terminal device 3, ... through the network. For example, after receiving the video data sent by the image acquisition device, the terminal device can store and display the video data (or the video data processed as required) to facilitate the user to monitor the monitoring area.

[0055] It can be understood that the image acquisition device and the terminal device can be in a one-to-one correspondence relationship, for example, terminal device 1 only obtains the video data sent by image acquisition device 1, terminal device 2 only obtains the video data sent by image acquisition device 2, and so on. The image acquisition device and the terminal device can also be in a many-to-many relationship, for example, terminal device 1 obtains the video data sent by image acquisition device 1, image acquisition device 2, image acquisition device 3, ...; terminal device 2 obtains the video data sent by image acquisition device 1, image acquisition device 2, image acquisition device 3, ...; terminal device 3 obtains the video data sent by image acquisition device 1, image acquisition device 2, image acquisition device 3, ...; and so on.

[0056] Understandably, Figure 1The application scenario shown is a video surveillance scenario. The embodiment of the present application can also be applied to any scenario that requires video (or image) privacy protection. The embodiment of the present application does not limit the application scenario.

[0057] It can be understood that image acquisition equipment can be deployed at intersections, school gates, shopping mall entrances and other locations.

[0058] It can be understood that in the embodiment of the present application, the actions of the image acquisition device to collect video data, transmit video data, and the terminal device to store, process and display video data are all carried out in compliance with the relevant data protection laws and policies of the country where they are located, and with the authorization given by the owner of the corresponding device.

[0059] In some embodiments, the image acquisition device may compress (or encode) the acquired video data to obtain a code stream, and then transmit the code stream to the terminal device to reduce the amount of transmitted data and reduce bandwidth requirements.

[0060] For example, reference Figure 2 , Figure 2 According to an embodiment of the present application, a schematic diagram of a compression framework is provided.

[0061] refer to Figure 2 The image acquisition device may include a camera, an encoding module (or encoder) and a sending module. The terminal device may include a display module, a decoding module (or decoder) and a receiving module.

[0062] Continue to refer Figure 2 , the process of the image acquisition device sending the original video data collected by the camera to the terminal device for display is as follows: the camera sends the collected original video data to the encoding module; the encoding module encodes the original video data to obtain a code stream, and sends the encoded code stream to the sending module; the sending module sends the code stream to the terminal device. The receiving module of the terminal device receives the code stream; transmits the code stream to the decoding module; the decoding module decodes the code stream to obtain reconstructed video data (i.e., video data processed according to requirements, such as video data processed for privacy areas), and sends the reconstructed video data to the display module, which displays the reconstructed video data.

[0063] For example, the following Figure 2 The image acquisition device is called the encoding end, and can Figure 2 The terminal device is called a decoding end; the encoding process of the encoding end and the decoding process of the decoding end can refer to the description of the subsequent embodiments.

[0064] It can be understood that in other embodiments, the system layer can also encrypt the encoded code stream and then store the encrypted code stream, or send it to the encoding end; before the decoding end receives the code stream, the system layer first decrypts the encrypted code stream and then sends it to the decoding end.

[0065] As mentioned above, the video data collected by the image acquisition device may carry the user's private information (such as face, ID information, license plate number, etc.). If the video data containing the user's private information is leaked, it may cause losses to the user. Therefore, the demand for video privacy protection technology is getting higher and higher. Video privacy protection is to make the image of the private area unrecognizable by coding, blocking or scrambling the private area in the video data, while the non-privacy area retains the original image content.

[0066] Currently, in order to solve the problem of leakage of video data carrying user privacy information, the encoding end usually inputs the privacy area image and the non-privacy area image of the video data into two encoders for encoding respectively, obtains a code stream of the privacy area image and a code stream of the non-privacy area image, and transmits them to the decoding end.

[0067] Specifically, it includes: before encoding the video data, the video data is first detected at the front end (such as a camera, or a preprocessing module newly added between the camera and the encoding module, etc.) to determine the privacy area and the non-privacy area, and the privacy area (such as a face, a license plate, etc.) and the non-privacy area (such as the background, etc.) in each frame image of the video data (hereinafter referred to as the original image) are separated to generate a privacy image and a non-privacy image.

[0068] For example, refer to Figure 3 Schematic diagram of privacy image and non-privacy image shown, Figure 3 (a) in the figure represents the original image; Figure 3 (b) in the figure represents a privacy image, and the non-privacy part of the privacy image is filled with a solid color (e.g., gray); Figure 3 (c) in the figure represents a non-private image, and the private part of the non-private image is filled with a solid color (eg, gray). It can be understood that the solid color filling is only used as an example, and other methods can also be used for filling, and this application does not impose a mandatory limitation on this.

[0069] At the encoding end, a video encoding and decoding scheme (such as H264, H265, AVS3, etc.) is used to perform independent encoding operations on private images and non-private images to obtain their respective corresponding bitstreams, namely, private image bitstreams and non-private image bitstreams.

[0070] At the decoding end, the private image code stream and the non-private image code stream are decoded respectively to obtain a reconstructed private image and a reconstructed non-private image. The reconstructed private image and the reconstructed non-private image are fused to obtain a reconstructed full image.

[0071] Figure 4 According to an embodiment of the present application, a schematic diagram of a dual-stream privacy protection method is shown. Figure 4 As shown, the method includes:

[0072] 1. Encoding preprocessing operations:

[0073] a) The front end of the image acquisition device performs front end detection on the original image to determine the privacy area and non-privacy area in the original image.

[0074] It is understandable that detection methods such as intelligent detection based on artificial intelligence (AI), such as regional convolutional neural network (R-CNN), can be used, and this application does not make specific limitations on this.

[0075] b) Generate a privacy image based on the detection results. The privacy area remains unchanged, and the pixels of the non-privacy area are replaced. The replacement method can be solid color filling, mosaic or Gaussian blur, etc., which is not limited in this application.

[0076] c) Generate a non-privacy image based on the detection result. The non-privacy area remains unchanged, and the pixels of the privacy area are replaced. The replacement method can be solid color filling, mosaic or Gaussian blur, etc., which is not limited in this application.

[0077] d) Inputting the private image and the non-private image into the encoder for encoding.

[0078] 2. Coding operation:

[0079] a) Send the private image to an encoder for encoding to generate a private image code stream.

[0080] b) Send the non-private image to another encoder for encoding to generate a non-private image code stream.

[0081] c) Output two code streams, namely, output the privacy image code stream and the non-privacy image code stream respectively.

[0082] d) Sending the private image code stream and the non-private image code stream to the decoder for decoding.

[0083] 3. Decoding operation:

[0084] a) Decode the non-private image code stream to obtain a non-private reconstructed image.

[0085] b) Decode the private image code stream to obtain the private reconstructed image.

[0086] c) The privacy-reconstructed image and the non-privacy-reconstructed image are fused to obtain a reconstructed full image. It can be understood that the reconstructed full image is consistent with the original image.

[0087] 4. System layer processing:

[0088] a) The encoded private image code stream is encrypted, while the non-private image code stream does not need to be encrypted.

[0089] b) If the decoding end wants to reconstruct the full image, the system layer needs to perform a decryption operation before decoding the private image code stream.

[0090] It can be understood that if the decryption operation is not performed, the decoding end cannot decode the private image stream, and the final image obtained is a non-private image, thereby achieving the purpose of protecting privacy.

[0091] It can be understood that in the above method, the privacy area image and the non-privacy area image are independently encoded and decoded, and the privacy area image and the non-privacy area image are encoded and decoded respectively, that is, the non-privacy area and the privacy area of ​​the original image are encoded and decoded twice, which increases the computational overhead of the encoding process and the decoding process.

[0092] In view of this, an embodiment of the present application provides a video data encoding method and a video data decoding method, in which an image only needs to be encoded and decoded once. Specifically, the image to be processed is divided into a plurality of image areas, wherein some image areas are privacy areas of the image to be processed, and some areas are non-privacy areas of the image to be processed. At the encoding end, each image area can be encoded to form the encoded data of the image to be processed, and transmitted to the decoding end, that is, it is transmitted in a single-frame image data packet. Then in the video data decoding method, the decoding end decodes the data sub-packets of each image area in the received image data packet.

[0093] In this way, only a single layer of encoding is performed on an image, and one code stream is generated accordingly (the transmission rate of each frame of image data packets transmitted by the same code stream is the same), which can reduce network bandwidth resources during transmission. In addition, each data area of ​​the image (including privacy areas and non-privacy areas) is only encoded and decoded once, which can reduce the computational overhead of the encoding and decoding processes.

[0094] For example, in some embodiments, in order to protect the privacy area in the image to be processed, each image area can be encoded at the encoding end based on the area where each image area is located in the image to be processed (for example, the privacy area or the non-privacy area) to obtain a privacy code stream and a non-privacy code stream; the privacy code stream and the non-privacy code stream are merged into one code stream as the data to be decoded and then transmitted / stored. At the decoding end, the user authority and the corresponding decoding strategy are determined based on the received data to be decoded. For example, corresponding to high user authority (first decoding authority), the image area in the non-privacy area is decoded based on the non-privacy code stream, and the image area in the privacy area is decoded based on the privacy code stream. For another example, corresponding to low user authority (second decoding authority), the image area in the non-privacy area is decoded only based on the non-privacy code stream, and the image area in the privacy area can be pixel filled with a preset pixel value.

[0095] It can be understood that the privacy area of ​​the reconstructed image obtained after low user authority decoding is filled with pixels using preset pixel values, thereby protecting the privacy area in the image to be processed.

[0096] In other embodiments, the privacy area of ​​the image to be processed may be masked to obtain a privacy masked image, and the privacy masked content in the privacy masked image may be encoded to obtain a privacy masked content code stream. In this way, at the decoding end in the above process, corresponding to low user authority, the image area in the privacy area may be decoded based on the privacy masked content code stream, rather than simply using a preset pixel value for pixel filling.

[0097] It can be understood that the masking processing of the privacy area can be mosaic, Gaussian blur, AI content replacement or superimposition of personalized content set by the user, which can make the masked content of the privacy area under the condition of low user authority more rich and personalized.

[0098] For example, reference Figure 5a The flowchart of the video data encoding method and the video data decoding method shown in FIG. Figure 5a As shown, the method includes:

[0099] 1. Encoding preprocessing operations:

[0100] a) The front end of the image acquisition device performs front end detection on the original image to determine the privacy area and non-privacy area in the original image.

[0101] It is understandable that detection methods such as intelligent detection based on artificial intelligence, such as regional convolutional neural networks, can be used, and this application does not make specific limitations on this.

[0102] Optionally, the front end of the image acquisition device may also perform masking processing on the privacy area to obtain a privacy masked image.

[0103] It can be understood that the masking process of the privacy area can be mosaic, Gaussian blur, AI content replacement or superimposition of personalized content set by the user, and this application does not make any specific limitation on this.

[0104] b) The original image is divided into multiple coding units (CUs) and input into the encoder for encoding.

[0105] It can be understood that a coding unit is generally a region with a size of 4×4 to 64×64, and a coding unit is only an example of an image region. In other embodiments, the image region may also be of other sizes. That is, one coding unit may be the image region mentioned above, or multiple coding units may constitute the image region mentioned above.

[0106] 2. Coding operation:

[0107] a) Based on the area where each coding unit is located in the original image, each coding unit is encoded to obtain a privacy code stream and a non-privacy code stream, that is, the privacy area of ​​the image to be processed is encoded to obtain a privacy code stream, and the non-privacy area of ​​the image to be processed is encoded to obtain a non-privacy code stream.

[0108] Optionally, the privacy masking content of the privacy masking area of ​​the privacy masking image may also be encoded to obtain a privacy masking content code stream. It can be understood that the privacy masking area of ​​the privacy masking image corresponds to the privacy area of ​​the image to be processed, and the non-privacy area of ​​the privacy masking image is consistent with the non-privacy area of ​​the image to be processed, so there is no need to encode the non-privacy area of ​​the privacy masking image.

[0109] It can be understood that the coded sub-data of the privacy area includes a privacy code stream and a privacy masking content code stream, and the coded sub-data of the non-privacy area includes a non-privacy code stream.

[0110] b) Interweaving the obtained privacy code stream, non-privacy code stream and privacy masking content code stream (optional) into one code stream (first encoded image data), and outputting or storing it.

[0111] It can be understood that the first encoded image data includes the encoded sub-data of the privacy area and the encoded sub-data of the non-privacy area.

[0112] In some embodiments, the privacy protection mode corresponding to the situation where a privacy code stream and a non-privacy code stream are obtained can be called a first mode, and the privacy protection mode corresponding to the situation where a privacy code stream, a non-privacy code stream and a privacy-masking content code stream are obtained can be called a second mode.

[0113] 3. Decoding operation:

[0114] a) receiving the first coded image data, and determining parameters such as user authority and coding unit division information (such as position information of each coding unit corresponding to the original image, etc.). For example, a user with low authority cannot obtain the content of the privacy area, while a user with high authority can obtain the content of the privacy area.

[0115] b) dividing the first coded image data into a plurality of coding units based on the coding unit division information, determining the region where each coding unit is located, obtaining corresponding coding unit sub-data (code stream) based on the region where each coding unit is located, and decoding each coding unit.

[0116] It can be understood that, corresponding to the first mode, since low user permissions cannot obtain the content of the privacy area, therefore, corresponding to low user permissions, only the coding unit in the non-privacy area is decoded based on the non-privacy code stream to obtain the decoded data of the non-privacy area of ​​the coding sub-data corresponding to the non-privacy area; the coding unit in the privacy area can use the preset pixel value (preset pixel value) to fill the pixel of the image corresponding to the privacy area to obtain the decoded data of the privacy area of ​​the coding sub-data corresponding to the privacy area. Since high user permissions can obtain the content of the privacy area, therefore, corresponding to high user permissions, the coding unit in the non-privacy area is decoded based on the non-privacy code stream, and the coding unit in the privacy area is decoded based on the privacy code stream to obtain the decoded data of the privacy area of ​​the coding sub-data corresponding to the privacy area.

[0117] Corresponding to the second mode, if the area where the coding unit is located is a non-privacy area, the coding unit is decoded based on the non-privacy code stream to obtain the decoded data of the non-privacy area of ​​the coded sub-data corresponding to the non-privacy area. If the area where the coding unit is located is a privacy area, and the user authority is a low user authority, the coding unit is decoded based on the privacy masking content code stream to obtain the decoded data of the privacy area of ​​the coded sub-data corresponding to the privacy area. If the area where the coding unit is located is a privacy area, and the user authority is a high authority, the coding unit is decoded based on the privacy code stream to obtain the decoded data of the privacy area of ​​the coded sub-data corresponding to the privacy area.

[0118] c) Based on the decoding results of each coding unit, image reconstruction is performed to obtain a reconstructed image (first decoded image data).

[0119] It can be understood that, corresponding to the first mode, each coding unit of the processed image is encoded and decoded only once; corresponding to the second mode, the coding unit of the non-privacy area is encoded only once, the coding unit of the privacy area is encoded twice, and each coding unit is decoded once; the computational overhead of the encoding process and the decoding process is reduced.

[0120] For example, Figure 5bAccording to an embodiment of the present application, a schematic diagram of an implementation flow of a video data encoding method is shown.

[0121] like Figure 5b As shown, the process includes but is not limited to the following steps:

[0122] S501: Acquire the first image in the video to be processed.

[0123] In some embodiments, the first image (referred to as the image to be processed) may be each frame of original image of the video to be processed (original video data).

[0124] S502: Divide the first image into a plurality of image regions, wherein the plurality of image regions include a first type of image regions and a second type of image regions.

[0125] In some embodiments, the plurality of image regions include a first type of image region (privacy region) and a second type of image region (non-privacy region). That is, any one of the plurality of image regions is located within the privacy region or within the non-privacy region. That is, in each image region, there are pixels within the privacy region or pixels within the non-privacy region.

[0126] S503: Encode the multiple image regions respectively to form first encoded image data, wherein the first encoded image data includes first region encoded data of the first type of image regions and second region encoded data of the second type of image regions.

[0127] In some embodiments, multiple image regions are respectively subjected to predictive coding to obtain first coded image data, wherein the predictive coding mode includes intra-frame prediction and inter-frame prediction.

[0128] In some embodiments, a plurality of encoded image regions located in a privacy area are packaged to form first region encoding data of a first type of image region (hereinafter referred to as a privacy code stream). A plurality of encoded coding units located in a non-privacy area are packaged to form second region encoding data of a second type of image region (hereinafter referred to as a non-privacy code stream).

[0129] S504: Send first encoded image data.

[0130] It can be understood that the first encoded image data can be sent in the first channel, wherein the first channel is a path for transmitting the first encoded image data.

[0131] In some embodiments, the first channel may be a wired channel, for example, wired transmission based on a display serial interface (DSI) protocol, optical fiber transmission, etc. In other embodiments, the first channel may also be a wireless channel, for example, wireless transmission based on a wireless fidelity (Wi-Fi) network, wireless transmission based on Bluetooth (BT), etc. This application does not impose specific restrictions on this.

[0132] In order to better understand the technical solutions of the embodiments of the present application, some technical solutions of the present application are introduced in detail below, taking the image area as the encoding unit as an example.

[0133] Embodiment 1

[0134] For example, Figure 6a According to an embodiment of the present application, a schematic diagram of a coding process is shown. It can be understood that Figure 6a The execution entities of each step of the process shown are the encoding end. Figure 6a The execution entities of each step will not be described repeatedly in the steps of the process shown.

[0135] like Figure 6a As shown, the process includes but is not limited to the following steps:

[0136] S601: Acquire an image to be processed.

[0137] In some embodiments, the image to be processed may be each frame of original image of original video data.

[0138] S602: Divide the image to be processed into multiple coding units.

[0139] In some embodiments, any one of the plurality of coding units is located within a privacy area or within a non-privacy area. That is, each coding unit is not divided by a boundary line of a privacy area. In one coding unit, all pixels are within the privacy area, or all pixels are within the non-privacy area.

[0140] S603: Perform predictive coding on multiple coding units to obtain a private code stream and a non-private code stream.

[0141] In some embodiments, multiple coding units are predictively encoded to obtain private code streams and non-private code streams, and the obtained division information, mode information, motion information, predictive coding mode, and filtering parameters are stored in a cache / memory. The predictive coding mode includes intra-frame prediction and inter-frame prediction.

[0142] In some embodiments, intra-frame prediction utilizes spatial pixel correlation within the current frame to remove information redundancy in the spatial domain. Specifically, when the current coding unit is intra-coded, the reconstructed pixels of the coding unit that is spatially adjacent to the current coding unit are referenced to predict the pixel value of the current coding unit. It can be understood that the coding unit that is spatially adjacent to the current coding unit can be referred to as a spatial domain position.

[0143] In some embodiments, inter-frame prediction utilizes the correlation between the current frame and the previously encoded / decoded frame in the time domain to remove information redundancy in the time domain. Specifically, when the current coding unit is inter-coded, the reconstructed pixels of the coding unit corresponding to the encoded / decoded frame are referenced, and the pixel value of the current coding unit is predicted by the motion offset. It can be understood that the coding unit corresponding to the encoded / decoded frame can be called a time domain position.

[0144] In some embodiments, multiple coding units in a privacy region are encapsulated to obtain a privacy network abstract layer unit (NALU), and the privacy NALUs form a privacy code stream. Multiple coding units in a non-privacy region are encapsulated to obtain a non-privacy NALU, and the non-privacy NALUs form a non-privacy code stream.

[0145] The types of NALU include: video coding layer network abstract layer unit (VCL NALU) and non-video coding layer network abstract layer unit (NON-VCL NALU). Among them, VCL NALU carries coded data (for example, the above-mentioned privacy NALU and non-privacy NALU are both VCL NALU), and NON-VCL NALU carries other information data.

[0146] For example, the predictive coding process can refer to Figure 6b The encoding process is shown in the schematic block diagram.

[0147] like Figure 6bAs shown, the predictive coding process includes: processing the input image to be processed in blocks to obtain coding units; subtracting the pixel value of the prediction block of the current coding unit (i.e., the coding unit corresponding to the current coding unit obtained by intra-frame prediction or inter-frame prediction) from the original pixel value of the current coding unit to obtain the pixel residual value (residual for short) of the current coding unit; then transforming the residual to obtain the transformation coefficient, so as to map the spatial domain residual to the frequency domain coefficient; then further reducing the size of the transformation coefficient by quantization; finally, entropy encoding the transformed quantized coefficient, motion information and control parameters (such as information entropy) to obtain the final compressed code stream, and the motion information includes motion vector (motion vector, MV) and the like. Exemplarily, a reconstruction operation can also be performed during the encoding process to generate a reconstructed image. Among them, the reconstructed image can be used as a reference image for other frames of images to be processed and / or a reference image for a privacy masked image.

[0148] Continue to refer to Figure 6b , the residual coefficients of the current coding unit can be inversely quantized and inversely transformed to obtain the residual of the current coding unit (wherein, the residual obtained by inverse quantization and inverse transformation is different from the residual before transformation and quantization, and this application does not distinguish between the two in terms of name). Secondly, the residual of the current coding unit is added to the prediction block of the current coding unit to obtain the reconstructed block of the current coding unit. Then, the reconstructed block of the current coding unit is loop-filtered to obtain the reconstructed block of the current coding unit after loop filtering and store it in the decoded image cache. In this way, the reconstructed blocks of all coding units obtained by dividing the image to be processed after loop filtering can form a reconstructed image.

[0149] It can be understood that the intra-frame prediction determines the prediction block of the current coding unit based on the residual obtained by inverse quantization and inverse transformation; the inter-frame prediction is determined based on the reconstructed image stored in the decoded image buffer.

[0150] In some embodiments, the non-privacy VCL NALU is encoded using entropy encoder 1, and the privacy VCL NALU is encoded using entropy encoder 2.

[0151] S604: Obtain a privacy masked image of the image to be processed.

[0152] In some embodiments, a privacy-masking image is obtained by performing masking processing on the privacy area.

[0153] It can be understood that the masking process of the privacy area can be mosaic, Gaussian blur, AI content replacement or superimposition of personalized content set by the user, and this application does not make any specific limitation on this.

[0154] S605: Encode the privacy-shielding content of the privacy-shielding image based on the coding information of the image to be processed to obtain a privacy-shielding content code stream.

[0155] In some embodiments, a reconstructed image, division information, mode information, motion information, prediction coding mode and filtering parameters obtained based on the original image are loaded. Using the division information, the coding unit where the privacy masking content is located is located in the coding order, and the coding unit is encoded using the entropy encoder 2, and the encoding result is encapsulated into the privacy masking content VCL NALU.

[0156] In some embodiments, encoding the encoding unit requires utilizing the following information:

[0157] a) When entropy coding of some syntax elements (such as the coding unit in the embodiment of the present application) is performed for context modeling, it is necessary to use syntax elements at adjacent positions. If the adjacent position is located in a non-privacy area, the syntax elements come from the data loaded in the previous step.

[0158] b) If it is intra prediction, the adjacent reconstructed pixels and the prediction coding mode need to be used. If the adjacent position is in a non-privacy area, the information comes from the data loaded in the previous step.

[0159] c) If it is inter-frame prediction, adjacent motion information needs to be used. If the adjacent position is located in a non-privacy area, the information comes from the data loaded in the previous step.

[0160] d) It is necessary to maintain (eg, update, modify, etc.) a list of motion information of the coded blocks (HMVP list) and related variables of quantization parameters (QP) according to the coding order.

[0161] e) Optionally, the privacy masking content coding unit is filtered using a filtering parameter.

[0162] It can be understood that in other embodiments, according to actual needs, the above Figure 6a The steps shown can be combined, deleted or replaced with other steps that are conducive to achieving the purpose of this application. For example, the above steps S601 and S602 can be combined into one step, and this application does not limit this.

[0163] For example, Figure 7 According to an embodiment of the present application, another schematic flow chart of the encoding process is shown. It can be understood that Figure 7 The execution entities of each step of the process shown are the encoding end. Figure 7 The execution entities of each step will not be described repeatedly in the steps of the process shown.

[0164] like Figure 7 As shown, the process includes but is not limited to the following steps:

[0165] S701: Acquire an image to be processed.

[0166] In some embodiments, the image to be processed may be each frame of original image of original video data.

[0167] S702: Divide the image to be processed into multiple coding units.

[0168] In some embodiments, any one of the plurality of coding units is located within a privacy area or within a non-privacy area. That is, each coding unit is not divided by a boundary line of a privacy area. In one coding unit, all pixels are within the privacy area, or all pixels are within the non-privacy area.

[0169] S703: Perform predictive coding on multiple coding units to obtain a private code stream and a non-private code stream.

[0170] In some embodiments, multiple coding units are predictively coded to obtain private code streams and non-private code streams, and the obtained division information, mode information, predictive coding mode, filtering parameters and other information are stored in a cache / memory. The predictive coding mode includes intra-frame prediction and inter-frame prediction.

[0171] In some embodiments, multiple coding units in a privacy area are encapsulated to obtain a privacy NALU, and the privacy NALUs constitute a privacy code stream. Multiple coding units in a non-privacy area are encapsulated to obtain a non-privacy NALU, and the non-privacy NALUs constitute a non-privacy code stream. Among them, the VCL NALU carries the coded data, and the NON-VCL NALU carries other information, such as privacy protection mode information, privacy protection area information, filtering parameters and other information.

[0172] For example, the specific process of predictive coding can refer to the above Figure 6b The encoding process schematic diagram and related description are shown in detail in this application.

[0173] When performing intra-frame mode derivation or inter-frame motion information derivation, if the current coding unit is in a non-privacy area and the reference position of the current coding unit is a privacy area, the reference position is set to be unavailable.

[0174] When performing intra-frame mode derivation or inter-frame motion information derivation, if the current coding unit is in a non-privacy area and the reference spatial domain position (corresponding to intra-frame prediction) or temporal domain position (corresponding to inter-frame prediction) is a privacy area, the reference spatial domain position or temporal domain position is set to unavailable.

[0175] When performing intra-frame mode derivation or inter-frame motion information derivation, if the current coding unit is in a privacy area (or a privacy masking area), after obtaining motion information, the obtained motion information cannot be added to the HMVP list.

[0176] When intra-frame prediction is performed on the current coding unit, if the current coding unit is in a non-privacy area and the reference pixel position is in a privacy area, the reference pixel is set to be unavailable.

[0177] When inter-frame prediction is performed on the current coding unit, if the current coding unit is in a non-privacy area (or a privacy masking area) and the reference pixel position is in the privacy area, the reference pixel is set to a preset pixel value.

[0178] The reference image is the first reconstructed image of the encoded frame before the current frame. The preset pixel value can be, for example, the median value or the value transmitted through the image header of the first image. As long as the preset pixel values ​​of the encoder and decoder are the same, the embodiment of the present application does not limit this.

[0179] When encoding multiple coding units, a reconstructed block of each coding unit in the multiple coding units can also be generated to obtain a first reconstructed image. The first reconstructed image is a reference image for other frames of images to be processed. The reconstructed pixels within the non-privacy area in the first reconstructed image are reconstructed pixels that have undergone loop filtering, and the reconstructed pixels within the privacy area are reconstructed pixels that have not undergone loop filtering. The reconstructed boundary pixels between the privacy area and the non-privacy area in the first reconstructed image are reconstructed pixels that have not undergone loop filtering. The reconstructed pixels of the privacy area (or privacy masking area) in the first reconstructed image can refer to the reconstructed pixels within the non-privacy area.

[0180] In some embodiments, the non-privacy VCL NALU is encoded using entropy encoder 1, and the privacy VCL NALU is encoded using entropy encoder 2.

[0181] S704: Obtain a privacy masked image of the image to be processed.

[0182] In some embodiments, a privacy-masking image is obtained by performing masking processing on the privacy area.

[0183] It can be understood that the masking process of the privacy area can be mosaic, Gaussian blur, AI content replacement or superimposition of personalized content set by the user, and this application does not make any specific limitations on this.

[0184] S705: Based on the coding information of the image to be processed, perform intra-frame prediction coding on the privacy-shielding content of the privacy-shielding image to obtain a privacy-shielding content bitstream.

[0185] In some embodiments, a reconstructed image, division information, mode information, prediction coding mode and filtering parameters obtained based on the original image are loaded. Using the division information, the coding unit where the privacy masking content is located is located in the coding order, and the coding unit is encoded using the entropy encoder 2, and the encoding result is encapsulated into the privacy masking content VCL NALU.

[0186] In some embodiments, encoding the encoding unit requires utilizing the following information:

[0187] a) When entropy coding of some syntax elements is used for context modeling, syntax elements at adjacent positions are needed. If the adjacent positions are located in a non-privacy area, the syntax elements come from the data loaded in the previous step.

[0188] b) If it is intra prediction, the adjacent reconstructed pixels and the prediction coding mode need to be used. If the adjacent position is in a non-privacy area, the information comes from the data loaded in the previous step.

[0189] d) The HMVP list and related variables of quantization parameters need to be maintained in the encoding order.

[0190] e) Optionally, the privacy masking content coding unit is filtered using a filtering parameter.

[0191] It can be understood that by constraining the privacy masking content encoding unit to only use intra-frame prediction for predictive encoding, the motion information data that needs to be saved after the first encoding pass is reduced.

[0192] It can be understood that in other embodiments, according to actual needs, the above Figure 7 The steps shown can be combined, deleted or replaced with other steps that are conducive to achieving the purpose of this application. For example, the above steps S701 and S702 can be combined into one step, and this application does not limit this.

[0193] It is understandable that the above Figure 6a and Figure 7 The flowcharts shown all obtain the privacy code stream, non-privacy code stream and privacy-masking content code stream through two encodings, that is, the privacy code stream and non-privacy code stream are obtained by the first encoding, and then the second encoding is performed based on the relevant encoding information stored in the first encoding to obtain the privacy-masking content code stream.

[0194] In some other embodiments, the privacy code stream, the non-privacy code stream and the privacy-shielding content code stream may also be obtained directly through one-pass encoding.

[0195] Specifically, corresponding to the second mode, based on the image to be processed and the privacy masked image, the non-privacy VCL NALU, the privacy VCL NALU and the privacy masked content VCL NALU are encoded. Among them, the non-privacy VCL NALU is encoded by entropy encoder 1, the privacy VCL NALU is encoded by entropy encoder 2, and the privacy masked content VCL NALU is encoded by entropy encoder 3.

[0196] It can be understood that, corresponding to the first mode, there is no need to obtain the privacy masking content code stream, and the encoding process includes: obtaining a non-privacy VCL NALU and a privacy VCL NALU based on the image to be processed, wherein the non-privacy VCL NALU is encoded by entropy encoder 1, and the privacy VCL NALU is encoded by entropy encoder 2.

[0197] For example, Figure 8 According to an embodiment of the present application, a flowchart of a decoding process is shown. It can be understood that Figure 8 The execution body of each step of the process shown is the decoding end. Figure 8 The execution entities of each step will not be described repeatedly in the steps of the process shown.

[0198] like Figure 8 As shown, the process includes but is not limited to the following steps:

[0199] S801: Obtain the encoded data to be processed.

[0200] In some embodiments, the decoding end obtains the coded data to be processed. In other embodiments, the decoding end may also obtain the coded data to be processed from a local memory. In other embodiments, the electronic device may also obtain the coded data to be processed from the cloud. This application is not limited to this.

[0201] S802: Determine user authority.

[0202] In some embodiments, the encoded data is parsed based on the obtained encoded data to be processed. For example, high-level syntax (such as SPS, PPS, PH, etc.) is parsed from the NON-VCL NALU, and user authority information is obtained, and the user authority is determined, such as low user authority or high user authority.

[0203] Optionally, the privacy protection area information can also be represented by the syntax of a coding unit, that is, after dividing into leaf nodes to obtain coding units, the privacy identifier is parsed to obtain the privacy identifier, which is used to indicate whether the coding unit is a privacy coding unit.

[0204] S803: Determine a corresponding decoding strategy based on user authority.

[0205] In some embodiments, corresponding to low user permissions, a low-privilege decoding strategy is used for decoding.

[0206] In other embodiments, corresponding to high user permissions, a high-authority decoding strategy is used for decoding.

[0207] For example, Figure 9a According to an embodiment of the present application, a schematic diagram of the implementation process of a low-authority decoding strategy is shown. It can be understood that Figure 9a The execution body of each step of the process shown is the decoding end. Figure 9a The execution entities of each step will not be described repeatedly in the steps of the process shown.

[0208] like Figure 9a As shown, the process includes but is not limited to the following steps:

[0209] S901: Obtaining information such as privacy protection mode information, privacy protection area information, and filtering parameters.

[0210] In some embodiments, the decoding end parses the NON-VCL NALU to obtain high-level syntax, such as a sequence parameter set (SPS), a picture parameter set (PPS), etc., and obtains privacy protection mode information (including a first mode and a second mode) and privacy protection area information.

[0211] It can be understood that in other embodiments, the privacy protection area information can also be represented by the syntax of the coding unit, that is, after dividing into leaf nodes to obtain the coding unit, the privacy identifier is parsed to obtain the privacy identifier, which is used to indicate whether the coding unit is a privacy coding unit.

[0212] Specifically, the decoding end uses entropy decoder 1 to parse the block header information (Slice / Tile / Patch Header) from the non-privacy VCL NALU; uses entropy decoder 1 to parse the filter parameters (such as SAO, ALF, etc.) from the non-privacy VCL NALU; and uses entropy decoder 1 to parse the partition information from the non-privacy VCL NALU.

[0213] S902: Parse the coding unit division information.

[0214] In some embodiments, the decoding end parses the coding unit division information and obtains each decoding unit, that is, divides the coded data to be processed into leaf nodes to obtain the coding units.

[0215] S903: Determine whether the divided area is completely within the privacy area.

[0216] In some embodiments, if the judgment result is yes, it means that the divided area (ie, the coding unit) is completely within the privacy area, and further judgment is needed whether it is the first mode or the second mode, then step S904 is executed to determine whether it is the privacy masking content mode.

[0217] In other embodiments, if the judgment result is no, it means that the divided area (i.e., the coding unit) is completely located in the non-privacy area and can be parsed based on the non-privacy code stream, then step S905 is executed to parse the internal information of the non-privacy masking unit based on the non-privacy code stream.

[0218] It can be understood that any coding unit is located in the privacy area or in the non-privacy area. That is, each coding unit is not divided by the boundary line of the privacy area. In a coding unit, all pixels are in the privacy area, or all pixels are in the non-privacy area.

[0219] S904: Determine whether it is a privacy shielding content mode.

[0220] In some embodiments, if the judgment result is yes, it indicates that it is the privacy shielding content mode (ie, the second mode), and parsing can be performed based on the privacy shielding content code stream, then step S906 is executed to parse the internal information of the privacy shielding unit based on the privacy shielding content code stream.

[0221] In some other embodiments, if the judgment result is no, it indicates the first mode, and the coding unit can be directly filled with a preset pixel value, then step S907 is executed to fill the privacy area with pixels.

[0222] S905: Parsing the internal information of the non-privacy unit based on the non-privacy code stream.

[0223] In some embodiments, the decoding end determines that the coding unit is located in the non-privacy area, and uses the entropy decoder 1 to parse the non-privacy VCL NALU to obtain the information (mode, residual, etc.) of the non-privacy coding unit.

[0224] It can be understood that when parsing non-privacy VCL NALU, the syntax elements of the privacy area position cannot be used for context modeling to select the context model. If the reference position is a privacy area, the reference position is set to be unavailable.

[0225] S906: Parsing the internal information of the privacy shielding unit based on the privacy shielding content code stream.

[0226] In some embodiments, when the decoding end determines that the coding unit is located in the privacy area and is in the second mode, it switches to entropy decoder 2 and parses the privacy masking content VCL NALU to obtain information (mode, residual, etc.) of the privacy masking content coding unit.

[0227] It can be understood that when parsing the privacy masking content VCL NALU, it is necessary to use the width information and height information of the privacy masking area corresponding to the privacy masking content parsed from the non-privacy VCL NALU. For example, the position information (including height information and width information) corresponding to the privacy masking area (or privacy area) is determined based on the position information of the non-privacy area; it is also possible to use the syntax elements of the non-privacy area position for context modeling, so the information parsed by the entropy decoder 1 from the non-privacy VCL NALU needs to be synchronized to the entropy decoder 2, so that the entropy decoder 2 can parse the privacy masking content coding unit from the privacy masking content VCL NALU.

[0228] In some embodiments, when parsing the privacy masking content VCL NALU, it is necessary to maintain variables related to the quantization parameter. For example, in the AVS3 and SVAC3 standards, the quantization parameter is NumDeltaQp. For another example, in the HEVC / H.265, VVC / H.266 and other standards, the quantization parameters are IsCuQpDeltaCoded and CuQpDeltaVal.

[0229] S907: Fill the privacy area with pixels.

[0230] In some embodiments, the decoding end determines that the coding unit is located in the privacy area and is in the first mode, and then uses a preset pixel value to fill the pixels of the coding unit.

[0231] S908: Reconstruct the image and perform filtering to obtain a first reconstructed image.

[0232] In some embodiments, the decoding end reconstructs the image based on the parsed information of the coding unit, and performs filtering based on the filtering parameters to obtain a first reconstructed image.

[0233] It can be understood that in other embodiments, according to actual needs, the above Figure 9a The steps shown can be combined, deleted or replaced with other steps that are conducive to achieving the purpose of this application. For example, the above steps S901 and S902 can be combined into one step, and this application does not limit this.

[0234] Figure 9b According to an embodiment of the present application, a schematic diagram of a data flow diagram of a decoding process is shown.

[0235] like Figure 9bAs shown, the decoding end obtains privacy protection mode information, privacy protection area information, filtering parameters and other information based on the non-privacy code stream, and parses the coding unit division information. Corresponding to the case where the divided area is completely located in the privacy area and is in the privacy masking content mode, the internal information of the privacy masking unit is parsed based on the privacy masking content code stream; corresponding to the case where the divided area is completely located in the privacy area and is not in the privacy masking content mode, pixel filling is performed; corresponding to the case where the divided area is not located in the privacy area, the internal information of the non-privacy unit is parsed based on the non-privacy code stream. The operation of reconstructing the image is performed and filtering is performed to obtain the first reconstructed image (not shown in the figure).

[0236] For example, Fig.10a According to the embodiment of the present application, a schematic diagram of the implementation process of a high-authority decoding strategy is shown. It can be understood that Fig.10a The execution body of each step of the process shown is the decoding end. Fig.10a The execution entities of each step will not be described repeatedly in the steps of the process shown.

[0237] like Fig.10a As shown, the process includes but is not limited to the following steps:

[0238] S1001: Obtaining information such as privacy protection mode, privacy protection area, and filtering parameters.

[0239] In some embodiments, the decoding end parses the coding unit division information and obtains each decoding unit, that is, divides the coded data to be processed into leaf nodes to obtain the coding units.

[0240] S1002: Parse the coding unit division information.

[0241] In some embodiments, the decoding end parses the coding unit division information and obtains each decoding unit, that is, divides the coded data to be processed into leaf nodes to obtain the coding units.

[0242] S1003: Determine whether the divided area is completely within the privacy area.

[0243] In some embodiments, if the judgment result is yes, it means that the divided area (ie, the coding unit) is completely located in the privacy area and can be parsed based on the privacy code stream, then step S1004 is executed to parse the internal information of the privacy unit based on the privacy code stream.

[0244] In other embodiments, if the judgment result is no, it means that the divided area (i.e., the coding unit) is completely located in the non-privacy area and can be parsed based on the non-privacy code stream, then step S1005 is executed to parse the internal information of the non-privacy masking unit based on the non-privacy code stream.

[0245] It can be understood that any coding unit is located in the privacy area or in the non-privacy area. That is, each coding unit is not divided by the boundary line of the privacy area. In a coding unit, all pixels are in the privacy area, or all pixels are in the non-privacy area.

[0246] S1004: Parsing internal information of the privacy unit based on the privacy code stream.

[0247] In some embodiments, when the decoding end determines that the coding unit is located in the privacy area, it switches to the entropy decoder 2 and parses the privacy VCL NALU to obtain the information (mode, residual, etc.) of the privacy coding unit.

[0248] It can be understood that when parsing the privacy VCL NALU, it is necessary to use the width and height information parsed from the non-privacy VCL NALU, and it is also possible to use the syntax elements at the non-privacy area position for context modeling. Therefore, the information parsed by entropy decoder 1 from the non-privacy VCL NALU needs to be synchronized to entropy decoder 2, so that entropy decoder 2 can parse the privacy coding unit information from the privacy VCL NALU.

[0249] In some embodiments, when parsing the privacy VCL NALU, it is necessary to maintain variables related to the quantization parameter. For example, in the AVS3 and SVAC3 standards, the quantization parameter is NumDeltaQp. For another example, in the HEVC / H.265, VVC / H.266 and other standards, the quantization parameters are IsCuQpDeltaCoded and CuQpDeltaVal.

[0250] S1005: Parsing the internal information of the non-privacy unit based on the non-privacy code stream.

[0251] In some embodiments, the decoding end determines that the coding unit is located in the non-privacy area, and uses the entropy decoder 1 to parse the non-privacy VCL NALU to obtain the information (mode, residual, etc.) of the non-privacy coding unit.

[0252] S1006: Reconstruct the image and perform filtering to obtain a second reconstructed image.

[0253] In some embodiments, the decoding end reconstructs the image based on the parsed information of the coding unit and performs filtering based on the filtering parameters to obtain a second reconstructed image.

[0254] It can be understood that in other embodiments, according to actual needs, the above Fig.10a The steps shown can be combined, deleted or replaced with other steps that are conducive to achieving the purpose of this application. For example, the above-mentioned step 9701 and step S902 can be combined into one step, and this application does not limit this.

[0255] Fig.10b According to an embodiment of the present application, a schematic diagram of a data flow diagram of a decoding process is shown.

[0256] like Fig.10b As shown, the decoding end obtains information such as privacy protection mode information, privacy protection area information, filtering parameters, etc. based on the non-privacy code stream, and parses the coding unit division information. Corresponding to the case where the divided area is completely located in the privacy area, the internal information of the privacy unit is parsed based on the privacy code stream; corresponding to the case where the divided area is not located in the privacy area, the internal information of the non-privacy unit is parsed based on the non-privacy code stream. The operation of reconstructing the image is performed and filtering is performed to obtain a second reconstructed image (not shown in the figure).

[0257] It can be understood that in the decoding process of the above embodiment, the information of each coding unit is obtained by first dividing into leaf nodes (coding units), determining the area where each coding unit is located, and then directly parsing based on the corresponding VCL NALU.

[0258] Embodiment 2

[0259] The specific encoding process can refer to the description of the encoding process in Example 1, and this application will not go into details here.

[0260] The decoding process is described below with reference to the accompanying drawings.

[0261] For example, Fig.11 According to an embodiment of the present application, another flowchart of a decoding process is shown. It can be understood that Fig.11 The execution body of each step of the process shown is the decoding end. Figure 1 The execution entities of each step will not be described repeatedly in the steps of the process shown.

[0262] like Fig.11 As shown, the process includes but is not limited to the following steps:

[0263] S1101: Obtain the encoded data to be processed.

[0264] In some embodiments, the decoding end obtains the coded data to be processed from the encoding end. In other embodiments, the decoding end may also obtain the coded data to be processed from a local memory. In other embodiments, the electronic device may also obtain the coded data to be processed from the cloud. This application does not limit this.

[0265] S1102: Determine user authority.

[0266] In some embodiments, the encoded data is parsed based on the obtained encoded data to be processed. For example, high-level syntax (such as SPS, PPS, PH, etc.) is parsed from the NON-VCL NALU, and user authority information is obtained, and user authority is determined, such as low user authority or high user authority.

[0267] S1103: Determine a corresponding decoding strategy based on user authority.

[0268] In some embodiments, corresponding to low user permissions, a low-privilege decoding strategy is used for decoding.

[0269] In other embodiments, corresponding to high user permissions, a high-authority decoding strategy is used for decoding.

[0270] For example, Fig.12a According to an embodiment of the present application, another implementation flow diagram of a low-authority decoding strategy is shown. It can be understood that Fig.12a The execution body of each step of the process shown is the decoding end. Fig.12a The execution entities of each step will not be described repeatedly in the steps of the process shown.

[0271] like Fig.12a As shown, the process includes but is not limited to the following steps:

[0272] S1201: Obtaining information such as privacy protection mode information, privacy protection area information, and filtering parameters.

[0273] In some embodiments, the decoder parses the NON-VCL NALU to obtain high-level syntax (such as SPS, PPS, PH, etc.), and obtains privacy protection mode information (including the first mode and the second mode) and privacy protection area information.

[0274] Specifically, the decoding end uses entropy decoder 1 to parse the block header information (Slice / Tile / Patch Header) from the non-privacy VCL NALU; uses entropy decoder 1 to parse the filter parameters (such as SAO, ALF, etc.) from the non-privacy VCL NALU; and uses entropy decoder 1 to parse the partition information from the non-privacy VCL NALU.

[0275] S1202: Parse the coding unit division information.

[0276] In some embodiments, the decoding end parses the coding unit division information and obtains each image region. For example, each image region may include multiple coding units.

[0277] S1203: Determine whether the divided area is completely within the privacy area.

[0278] In some embodiments, if the judgment result is yes, it means that the divided area (ie, the coding unit) is completely within the privacy area, and further judgment is needed whether it is the first mode or the second mode, then step S1206 is executed to determine whether it is the privacy masking content mode.

[0279] In other embodiments, if the judgment result is no, it means that the divided area (i.e., the coding unit) is completely located in the non-privacy area, and it is necessary to further determine whether it has been divided into a leaf node, then step S1204 is executed to determine whether it has been divided into a leaf node.

[0280] S1204: Determine whether the node has been divided into leaf nodes.

[0281] In some embodiments, if the judgment result is yes, it means that it has been divided into leaf nodes and can be directly parsed based on the non-private code stream, then step S1205 is executed to parse the internal information of the non-private unit based on the non-private code stream.

[0282] In other embodiments, if the judgment result is no, it means that the division has not been completed to a leaf node, and the division needs to be continued after parsing the division information, then step S1202 is executed to parse the coding unit division information.

[0283] S1205: Parsing the internal information of the non-privacy unit based on the non-privacy code stream.

[0284] In some embodiments, the decoding end determines that the coding unit is located in the non-privacy area, and uses the entropy decoder 1 to parse the non-privacy VCL NALU to obtain the information (mode, residual, etc.) of the non-privacy coding unit.

[0285] S1206: Determine whether it is a privacy shielding content mode.

[0286] In some embodiments, if the judgment result is yes, it indicates that it is a privacy masking content mode (ie, the second mode), and further judgment is needed as to whether it has been divided into a leaf node, then step S1208 is executed to judge whether it has been divided into a leaf node.

[0287] In some other embodiments, if the judgment result is no, it indicates the first mode, and the coding unit can be directly filled with a preset pixel value, then step S1207 is executed to fill the privacy area with pixels.

[0288] S1207: Fill the privacy area with pixels.

[0289] In some embodiments, the decoding end determines that the coding unit is located in the privacy area and is in the first mode, and then uses a preset pixel value to fill the pixels of the coding unit.

[0290] S1208: Determine whether the node has been divided into leaf nodes.

[0291] In some embodiments, if the judgment result is yes, it means that it has been divided into leaf nodes and can be directly parsed based on the privacy shielding content code stream, then step S1210 is executed to parse the internal information of the privacy shielding unit based on the privacy shielding content code stream.

[0292] In other embodiments, if the judgment result is no, it means that the data has not been divided into leaf nodes and it is necessary to parse the division information and continue the division, then step S1209 is executed to parse the division information and perform the division based on the privacy masking content code stream.

[0293] S1209: Parse the partition information based on the privacy masking content code stream and perform partitioning.

[0294] In some embodiments, if the decoding end determines that the data has not been divided into leaf nodes, it parses the division information based on the privacy masking content code stream, and divides the data into leaf nodes based on the division information, and then executes step S1208 to determine whether the data has been divided into leaf nodes.

[0295] S1210: Parsing internal information of the privacy shielding unit based on the privacy shielding content code stream.

[0296] In some embodiments, when the decoding end determines that the coding unit is located in the privacy area and is in the second mode, it switches to entropy decoder 2 and parses the privacy masking content VCL NALU to obtain information (mode, residual, etc.) of the privacy masking content coding unit.

[0297] S1211: Reconstruct the image and perform filtering to obtain a first reconstructed image.

[0298] In some embodiments, the decoding end reconstructs the image based on the parsed information of the coding unit and performs filtering based on the filtering parameters to obtain a first reconstructed image.

[0299] It can be understood that in other embodiments, according to actual needs, the above Fig.12a The steps shown can be combined, deleted or replaced with other steps that are conducive to achieving the purpose of this application. For example, the above steps S1201 and S1202 can be combined into one step, and this application does not limit this.

[0300] Figure 12b According to an embodiment of the present application, a schematic diagram of a data flow diagram of a decoding process is shown.

[0301] like Figure 12bAs shown, the decoding end obtains privacy protection mode information, privacy protection area information, filtering parameters and other information based on the non-privacy code stream, and parses the division information. Corresponding to the situation where the divided area is completely located in the privacy area, is in the privacy masking content mode and has been divided into leaf nodes, the internal information of the privacy masking unit is parsed based on the privacy masking content code stream. Corresponding to the situation where the divided area is completely located in the privacy area, is in the privacy masking content mode and has not been divided into leaf nodes, the division information is parsed based on the privacy masking content code stream and continues to determine whether it has been divided into leaf nodes after division. Corresponding to the situation where the divided area is completely located in the privacy area and is not in the privacy masking content mode, pixel filling is performed. Corresponding to the situation where the divided area is not located in the privacy area and has been divided into leaf nodes, the internal information of the non-privacy unit is parsed based on the non-privacy code stream. Corresponding to the situation where the divided area is not located in the privacy area and has not been divided into leaf nodes, the division information is parsed and divided. The operation of reconstructing the image is performed and filtering is performed to obtain a first reconstructed image (not shown in the figure).

[0302] For example, Fig.13a According to the embodiment of the present application, a schematic diagram of the implementation process of a high-authority decoding strategy is shown. It can be understood that Fig.13a The execution body of each step of the process shown is the decoding end. Fig.13a The execution entities of each step will not be described repeatedly in the steps of the process shown.

[0303] like Fig.13a As shown, the process includes but is not limited to the following steps:

[0304] S1301: Obtaining information such as privacy protection mode information, privacy protection area information, and filtering parameters.

[0305] In some embodiments, the decoder parses the NON-VCL NALU to obtain high-level syntax (such as SPS, PPS, PH, etc.), and obtains privacy protection mode information (including the first mode and the second mode) and privacy protection area information.

[0306] Specifically, the decoding end uses entropy decoder 1 to parse the block header information (Slice / Tile / Patch Header) from the non-privacy VCL NALU; uses entropy decoder 1 to parse the filter parameters (such as SAO, ALF, etc.) from the non-privacy VCL NALU; and uses entropy decoder 1 to parse the partition information from the non-privacy VCL NALU.

[0307] S1302: Parse the coding unit division information.

[0308] In some embodiments, the decoding end parses the coding unit division information and obtains each decoding unit, that is, divides the coded data to be processed into leaf nodes to obtain the coding units.

[0309] S1303: Determine whether the divided area is completely within the privacy area.

[0310] In some embodiments, if the judgment result is yes, it means that the divided area (ie, the coding unit) is completely located in the privacy area, and further judgment is needed as to whether it has been divided into a leaf node, then step S1306 is executed to judge whether it has been divided into a leaf node.

[0311] In other embodiments, if the judgment result is no, it means that the divided area (i.e., the coding unit) is completely located in the non-privacy area, and it is necessary to further determine whether it has been divided into a leaf node, then step S1304 is executed to determine whether it has been divided into a leaf node.

[0312] S1304: Determine whether the node has been divided into leaf nodes.

[0313] In some embodiments, if the judgment result is yes, it means that it has been divided into leaf nodes and can be directly parsed based on the non-private code stream, then step S1305 is executed to parse the internal information of the non-private unit based on the non-private code stream.

[0314] In other embodiments, if the judgment result is no, it means that the division has not been completed to a leaf node, and the division needs to be continued after parsing the division information, then step S1302 is executed to parse the coding unit division information.

[0315] S1305: Parsing the internal information of the non-privacy unit based on the non-privacy code stream.

[0316] In some embodiments, the decoding end determines that the coding unit is located in the non-privacy area, and uses the entropy decoder 1 to parse the non-privacy VCL NALU to obtain the information (mode, residual, etc.) of the non-privacy coding unit.

[0317] S1306: Determine whether the node has been divided into leaf nodes.

[0318] In some embodiments, if the judgment result is yes, it means that it has been divided into leaf nodes and can be directly parsed based on the privacy code stream, then step S1308 is executed to parse the internal information of the privacy unit based on the privacy code stream.

[0319] In other embodiments, if the judgment result is no, it means that the partition has not been made to a leaf node, and the partition information needs to be parsed before continuing the partition, then step S1307 is executed to parse the partition information based on the privacy code stream and perform the partition.

[0320] S1307: Parse the partition information based on the privacy code stream and perform partitioning.

[0321] In some embodiments, if the decoding end determines that the data has not been divided into leaf nodes, it parses the division information based on the privacy code stream, divides the data into leaf nodes based on the division information, and then executes step S1306 to determine whether the data has been divided into leaf nodes.

[0322] S1308: Parsing internal information of the privacy unit based on the privacy code stream.

[0323] In some embodiments, when the decoding end determines that the coding unit is located in the privacy area and is in the second mode, it switches to entropy decoder 2 and parses the privacy masking content VCL NALU to obtain information (mode, residual, etc.) of the privacy masking content coding unit.

[0324] S1309: reconstruct the image and perform filtering to obtain a second reconstructed image.

[0325] In some embodiments, the decoding end reconstructs the image based on the information of the parsed coding unit and performs filtering based on the filtering parameters to obtain a second reconstructed image. It can be understood that in other embodiments, according to actual needs, the above Fig.13a The steps shown can be combined, deleted or replaced with other steps that are conducive to achieving the purpose of this application. For example, the above steps S1301 and S1302 can be combined into one step, and this application does not limit this.

[0326] Fig.13b According to an embodiment of the present application, a schematic diagram of a data flow diagram of a decoding process is shown.

[0327] like Fig.13b As shown, the decoding end obtains privacy protection mode information, privacy protection area information, filtering parameters and other information based on the non-privacy code stream, and parses the division information. Corresponding to the case where the divided area is completely located in the privacy area and has been divided into leaf nodes, the internal information of the privacy unit is parsed based on the privacy code stream. Corresponding to the case where the divided area is completely located in the privacy area and has not been divided into leaf nodes, the division information is parsed based on the privacy code stream and continues to determine whether it has been divided into leaf nodes after division. Corresponding to the case where the divided area is not located in the privacy area and has been divided into leaf nodes, the internal information of the non-privacy unit is parsed based on the non-privacy code stream. Corresponding to the case where the divided area is not located in the privacy area and has not been divided into leaf nodes, the division information is parsed and divided. The operation of reconstructing the image is performed and filtering is performed to obtain a second reconstructed image (not shown in the figure).

[0328] It can be understood that the decoding process of the above embodiment is to first divide the image area into image areas, determine the area where each image area is located, and then divide each image area into leaf nodes (coding units), and obtain the information of each coding unit based on the corresponding VCL NALU analysis.

[0329] In order to better understand the technical solutions of the embodiments of the present application, the structure of the electronic device involved in the present application is introduced below with reference to the accompanying drawings.

[0330] For example, Fig.14 According to an embodiment of the present application, a structural schematic diagram of an electronic device is shown.

[0331] like Fig.14 As shown, the electronic device may include a processor 110, a memory 120, an interface module 130, a power module 140, a wireless communication module 150, a mobile communication module 160, an audio module 170, a sensor module 180, a button 190, a camera 191, and a display screen 192, etc.

[0332] It is to be understood that the structure illustrated in the embodiments of the present application does not constitute a specific limitation on the electronic device. In other embodiments of the present application, the electronic device may include more or fewer components than shown in the figure, or combine certain components, or split certain components, or arrange the components differently. The components shown in the figure may be implemented in hardware, software, or a combination of software and hardware.

[0333] The processor 110 may include one or more processing units, for example, the processor 110 may include an application processor (application processor, AP), a modem processor, a GPU, an image signal processor (image signal processor, ISP), a controller, a video codec, a digital signal processor (digital signal processor, DSP), a baseband processor, and / or a neural network processor (neural-network processing unit, NPU), etc. Among them, different processing units can be independent devices or integrated in one or more processors. The processor 110 can be used to execute the video data encoding method and the video data decoding method provided in the embodiments of the present application.

[0334] The controller can generate operation control signals according to the instruction operation code and timing signal to complete the control of instruction fetching and execution.

[0335] The processor 110 may also be provided with a memory for storing instructions and data. In some embodiments, the memory in the processor 110 is a cache memory. The memory may store instructions or data that the processor 110 has just used or cyclically used. If the processor 110 needs to use the instruction or data again, it may be directly called from the memory. This avoids repeated access, reduces the waiting time of the processor 110, and thus improves the efficiency of the system.

[0336] In some embodiments, the processor 110 may include one or more interfaces. The interface may include an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a subscriber identity module (SIM) interface, and / or a universal serial bus (USB) interface, etc.

[0337] The power module 140 is used to receive charging input from the charger and to power the processor 110 , the memory 120 , the display screen 192 , and the wireless communication module 150 . In some other embodiments, the power module 140 may also be disposed in the processor 110 .

[0338] The wireless communication module 150 can provide wireless communication solutions including wireless local area networks (WLAN) (such as wireless fidelity networks), Bluetooth, global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC), infrared (IR), etc., which are applied to the terminal device 100. The wireless communication module 150 can be one or more devices integrating at least one communication processing module.

[0339] The mobile communication module 160 may provide solutions for wireless communications including 2G / 3G / 4G / 5G, etc., applied to the terminal device 100. The mobile communication module 160 may include at least one filter, a switch, a power amplifier, a low noise amplifier (LNA), etc.

[0340] The electronic device implements the display function through a GPU, a display screen 192, and an application processor. The GPU is a microprocessor for image processing, which connects the display screen 192 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. The processor 110 may include one or more GPUs that execute program instructions to generate or change display information.

[0341] The display screen 192 is used to display images. The display screen 192 includes a display panel. The display panel can be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light emitting diode or an active-matrix organic light emitting diode (AMOLED), etc. In some embodiments, the electronic device 100 may include 1 or N display screens 192, where N is a positive integer greater than 1. In some embodiments, the display screen 192 can display a first reconstructed image and a second reconstructed image, etc.

[0342] The electronic device can realize the shooting function through ISP, camera 193, video codec, GPU, display screen 194 and application processor.

[0343] ISP is used to process the data fed back by camera 193. For example, when taking a photo, the shutter is opened, and the light is transmitted to the camera photosensitive element through the lens. The light signal is converted into an electrical signal, and the camera photosensitive element transmits the electrical signal to ISP for processing and converts it into an image visible to the naked eye. ISP can also perform algorithm optimization on the noise, brightness, and skin color of the image. ISP can also optimize the exposure, color temperature and other parameters of the shooting scene. In some embodiments, ISP can be set in camera 193.

[0344] Camera 193 is used to capture still images or videos. The object generates an optical image through the lens and projects it onto the photosensitive element. The photosensitive element can be a charge coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the optical signal into an electrical signal, and then passes the electrical signal to the ISP for conversion into a digital image signal. The ISP outputs the digital image signal to the DSP for processing. The DSP converts the digital image signal into an image signal in a standard RGB, YUV or other format. In some embodiments, the electronic device may include 1 or N cameras 193, where N is a positive integer greater than 1. In some embodiments, the electronic device obtains the image to be processed through the camera 193.

[0345] Digital signal processors are used to process digital signals. In addition to processing digital image signals, they can also process other digital signals.

[0346] Video codecs are used to compress or decompress digital videos. An electronic device may support one or more video codecs. Thus, the electronic device 100 may play or record videos in a variety of coding formats, such as moving picture experts group (MPEG) 1, MPEG2, MPEG3, MPEG4, etc.

[0347] NPU is a neural network (NN) computing processor. By drawing on the structure of biological neural networks, such as the transmission mode between neurons in the human brain, it can quickly process input information and can also continuously self-learn. Through NPU, applications such as intelligent cognition of electronic device 100 can be realized, such as image recognition, face recognition, voice recognition, text understanding, etc.

[0348] The memory 120 can be used to store computer executable program codes, which include instructions. The memory 120 may include a program storage area and a data storage area. Among them, the program storage area may store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc. The data storage area may store data created during the use of the electronic device 100 (such as audio data, a phone book, etc.), etc. In addition, the memory 120 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, a universal flash storage (universal flash storage, UFS), etc. The processor 110 executes various functional applications and data processing of the terminal device 100 by running instructions stored in the memory 120, and / or instructions stored in a memory provided in the processor. In some implementation examples, the processor 110 executes the video data encoding method and the video data decoding method provided in the embodiment of the present application by running instructions stored in the memory 120.

[0349] The electronic device 100 can implement audio functions through the audio module 170 and an application processor, etc.

[0350] The sensor module 180 may include a pressure sensor, a gyro sensor, an air pressure sensor, a magnetic sensor, an acceleration sensor, a distance sensor, a proximity light sensor, a fingerprint sensor, a temperature sensor, a touch sensor, an ambient light sensor, a bone conduction sensor, and the like.

[0351] The key 190 includes a power key, a volume key, etc. The key 190 may be a mechanical key or a touch key. The terminal device 100 may receive key input and generate key signal input related to user settings and function control of the terminal device 100.

[0352] Understandably, Fig.14 The structure of the electronic device 100 shown is only an example. In other embodiments, the electronic device 100 may include more or fewer components than shown, or combine some components, or separate some components, or arrange the components differently. The components shown in the figure may be implemented in hardware, software, or a combination of software and hardware.

[0353] It can be understood that in some embodiments, the encoding end and the decoding end can be two electronic devices. Fig.14 The electronic device shown can be an encoding end or a decoding end. In other embodiments, the encoding end and the decoding end can also be in the same electronic device, such as Fig.14 The electronic device shown in the figure is not limited in this application.

[0354] The embodiment of the present application also provides a program product, which includes instructions. When the instructions are executed by an electronic device, the electronic device can implement the video data encoding method and the video data decoding method provided by the embodiment of the present application.

[0355] An embodiment of the present application also provides a chip, the chip system includes a processing circuit and a storage medium, and the storage medium stores computer program code; when the computer program code is executed by the processing circuit, the video data encoding method and the video data decoding method provided in the embodiment of the present application are implemented.

[0356] The various embodiments of the mechanism disclosed in the present application can be implemented in hardware, software, firmware or a combination of these implementation methods. The embodiments of the present application can be implemented as a computer program or program code executed on a programmable system, which includes at least one processor, a storage system (including volatile and non-volatile memory and / or storage elements), at least one input device and at least one output device.

[0357] Program code may be applied to input instructions to perform the functions described herein and to generate output information. The output information may be applied to one or more output devices in a known manner. For purposes of this application, a processing system includes any system having a processor such as, for example, a digital signal processor (DSP), a microcontroller, an application specific integrated circuit (ASIC), or a microprocessor.

[0358] Program code can be implemented with high-level programming language or object-oriented programming language to communicate with the processing system. When necessary, program code can also be implemented with assembly language or machine language. In fact, the mechanism described in this application is not limited to the scope of any specific programming language. In either case, the language can be a compiled language or an interpreted language.

[0359] In some cases, the disclosed embodiments may be implemented in hardware, firmware, software, or any combination thereof. The disclosed embodiments may also be implemented as instructions carried or stored on one or more temporary or non-temporary machine-readable (e.g., computer-readable) storage media, which may be read and executed by one or more processors. For example, instructions may be distributed over a network or through other computer-readable media. Therefore, a machine-readable medium may include any mechanism for storing or transmitting information in a machine (e.g., computer) readable form, including, but not limited to, a floppy disk, an optical disk, an optical disk, a magneto-optical disk, a read-only memory (ROM), a random access memory (RAM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), a magnetic card or an optical card, a flash memory, or a tangible machine-readable memory for transmitting information (e.g., a carrier wave, an infrared signal, a digital signal, etc.) using the Internet in an electrical, optical, acoustic, or other form of propagation signal. Accordingly, machine-readable media include any type of machine-readable media suitable for storing or transmitting electronic instructions or information in a form readable by a machine (eg, a computer).

[0360] In the accompanying drawings, some structural or method features may be shown in a specific arrangement and / or order. However, it should be understood that such a specific arrangement and / or order may not be required. Instead, in some embodiments, these features may be arranged in a manner and / or order different from that shown in the illustrative drawings. In addition, the inclusion of structural or method features in a particular figure does not mean that such features are required in all embodiments, and in some embodiments, these features may not be included or may be combined with other features.

[0361] It should be noted that the units / modules mentioned in the various device embodiments of the present application are all logical units / modules. Physically, a logical unit / module can be a physical unit / module, or a part of a physical unit / module, or can be implemented as a combination of multiple physical units / modules. The physical implementation method of these logical units / modules themselves is not the most important. The combination of functions implemented by these logical units / modules is the key to solving the technical problems proposed by the present application. In addition, in order to highlight the innovative part of the present application, the above-mentioned device embodiments of the present application do not introduce units / modules that are not closely related to solving the technical problems proposed by the present application, which does not mean that there are no other units / modules in the above-mentioned device embodiments.

[0362] It should be noted that in the examples and description of this patent, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of more restrictions, an element defined by the sentence "comprises one" does not exclude the existence of other identical elements in the process, method, article or device including the element.

[0363] Although the present application has been illustrated and described with reference to certain preferred embodiments thereof, it will be apparent to those skilled in the art that various changes in form and details may be made therein without departing from the spirit and scope of the present application.

Claims

1. A video data encoding method, characterized in that: include: Obtaining the first image in the video to be processed; dividing the first image into a plurality of image regions, wherein the plurality of image regions include a first type of image region and a second type of image region, wherein the image regions are coding units; Encoding the plurality of image regions respectively to form first encoded image data, wherein the first encoded image data includes first region encoded data of the first type of image regions and second region encoded data of the second type of image regions; The first encoded image data is transmitted.

2. The method according to claim 1, characterized in that: The first type of image area is a privacy area, and the second type of image area is a non-privacy area.

3. The method according to claim 2, characterized in that The first region encoding data is obtained by encoding the first type of image region in the following manner: performing masking processing on the first type of image area of ​​the first image to obtain a first type of masked area corresponding to the first type of image area; The first-category image region and the first-category masked region are encoded respectively to obtain the first-region encoding data, wherein the first-region encoding data includes first-category image region encoding sub-data and first-category masked region encoding sub-data.

4. The method according to claim 3, characterized in that The masking process includes at least one of the following: Mosaic filling, Gaussian blur, AI content replacement, and overlay of user-set content.

5. The method according to claim 1, characterized in that The sending of the first encoded image data comprises: The first encoded image data is transmitted through a first channel.

6. A video data decoding method, characterized in that: include: Receiving first encoded image data of a first image in a video to be processed, wherein the first encoded image data includes first region encoding data of a first type of image region of the first image and second region encoding data of a second type of image region, the image region being an encoding unit; determining a privacy protection mode of the first encoded image data, wherein the privacy protection mode comprises a first mode and a second mode; The first encoded image data is decoded based on the privacy protection mode to obtain first decoded image data.

7. The method according to claim 6, characterized in that The first type of image area is a privacy area, and the second type of image area is a non-privacy area; and The first region coded data includes first-type image region coded sub-data and first-type masked region coded sub-data, wherein the first-type masked region coded sub-data is obtained by encoding the first-type image region after masking processing.

8. The method according to claim 7, characterized in that The privacy protection mode is a first mode, and The decoding the first encoded image data based on the privacy protection mode to obtain first decoded image data includes: corresponding to the first coded image data, having a first decoding authority, decoding the first-category image region coded sub-data to obtain first region decoded data corresponding to the first region coded data; Corresponding to the first encoded image data, having a second decoding authority, the image area corresponding to the first area encoded data in the first image is set to a preset pixel value to obtain first area decoded data corresponding to the first area encoded data, wherein the first decoding authority is higher than the second decoding authority.

9. The method according to claim 7, characterized in that: The privacy protection mode is the second mode, and The decoding the first encoded image data based on the privacy protection mode to obtain first decoded image data includes: corresponding to the first coded image data, having a first decoding authority, decoding the first-category image region coded sub-data to obtain first region decoded data corresponding to the first region coded data; Corresponding to the first encoded image data having a second decoding authority, the first type of masked area encoded sub-data is decoded to obtain first area decoded data corresponding to the first area encoded data, wherein the first decoding authority is higher than the second decoding authority.

10. The method according to claim 7, characterized in that The privacy protection mode is the first mode or the second mode, and The decoding the first encoded image data based on the privacy protection mode to obtain first decoded image data includes: The second region coded data is decoded to obtain second region decoded data corresponding to the second region coded data.

11. The method according to claim 6, characterized in that Also includes: Receiving region division information of the first encoded image data; Based on the region division information, region coding sub-data of each image region in the first image is determined.

12. A readable medium, characterized in that The readable medium stores instructions, which, when executed on an electronic device, enable the electronic device to execute the method according to any one of claims 1 to 11.

13. An electronic device, characterized in that: include: a memory for storing instructions to be executed by one or more processors of the electronic device, and The processor is one of the processors of the electronic device, and is used to execute the method according to any one of claims 1 to 11.

14. A chip, characterized in that: The chip system includes a processing circuit and a storage medium, wherein the storage medium stores computer program code; when the computer program code is executed by the processing circuit, the method according to any one of claims 1 to 11 is implemented.

Images