Video processing method, device, computer equipment and storage medium

Abstract

The present disclosure provides a video processing method, apparatus, computer equipment and storage medium, wherein the method comprises: obtaining a video to be processed; identifying at least one type of target object in each frame of the video to be processed, wherein each type of target object is related to personal information; blurring the target object in the identified video image to obtain a target image; and generating a blurred target video based on the target image corresponding to each frame of the video image.

Description

Technical Field

[0001] The present disclosure relates to the field of image processing technology, and in particular to a video processing method, apparatus, computer equipment and storage medium. Background Art

[0002] With the rapid development of network technology, people can obtain a variety of information through various channels and methods, such as video information, picture information, etc. Although the diversification of information acquisition channels and methods has increased the convenience of information acquisition, it has also increased the risk of information leakage. For example, the leakage of sensitive information such as license plate information and face information in driving videos affects data security.

[0003] How to prevent the leakage of sensitive information in video information and ensure data security while ensuring the convenience of information acquisition has become an urgent problem that needs to be solved. Summary of the invention

[0004] The embodiments of the present disclosure at least provide a video processing method, apparatus, computer equipment and storage medium.

[0005] In a first aspect, an embodiment of the present disclosure provides a video processing method, including:

[0006] Get the video to be processed;

[0007] Identify at least one type of target object in each frame of the video to be processed, each type of target object being related to personal information;

[0008] Performing blur processing on the identified target object in the video image to obtain a target image;

[0009] Based on the target images corresponding to each frame of the video image, a blurred target video is generated.

[0010] Here, the target objects related to personal information are usually sensitive information in video images. By identifying various types of target objects related to personal information in each frame of video images and then blurring the target objects in each frame of video images, it is possible to remove sensitive information from each frame of video images, and then obtain a target video with various sensitive information removed, that is, a desensitized target video, which effectively improves the security of video data.

[0011] In a possible implementation manner, the step of performing blur processing on the identified target object in the video image to obtain the target image includes:

[0012] In response to identifying a plurality of the target objects in the video image, extracting an initial sub-image corresponding to each of the target objects from the video image;

[0013] Performing fuzzy processing on each of the initial sub-images to obtain a target sub-image corresponding to each of the initial sub-images;

[0014] Each of the target sub-images is used to replace the corresponding initial sub-image in the video image to obtain the target image.

[0015] This implementation, by extracting the initial sub-image corresponding to each target object and blurring the initial sub-image, can improve the efficiency of the overall blurring process compared to directly blurring the entire video image.

[0016] In a possible implementation manner, for any of the initial sub-images, blur processing is performed on the initial sub-image according to the following steps:

[0017] Dividing the initial sub-image into a plurality of processing regions;

[0018] Determine a target pixel value corresponding to each processing area based on the pixel value of each pixel point in each processing area;

[0019] The pixel value corresponding to each pixel point in each processing area is replaced with the determined target pixel value to obtain the target sub-image corresponding to the initial sub-image.

[0020] In this embodiment, the pixel values ​​of each pixel in different processing areas are different, and the target pixel value corresponding to each processing area is also different. By using different target pixel values ​​to replace the pixel values ​​of multiple processing areas, there are differences in pixel values ​​between the various processing areas in the target image after replacement, that is, there are changes in color, so that the replaced target image is more natural.

[0021] In a possible implementation, determining the target pixel value corresponding to each processing area based on the pixel values ​​corresponding to each pixel point in each processing area includes:

[0022] Based on the pixel values ​​of each pixel point in each of the processing areas, the average pixel value corresponding to each of the processing areas is determined, and the average pixel value corresponding to each of the processing areas is used as the target pixel value corresponding to the processing area.

[0023] In this implementation, the pixel mean is used as the target pixel value, so that the target pixel value can represent the concentration degree of each pixel point in the corresponding processing area, thereby making the pixel values ​​between each processing area more balanced after the pixel values ​​are replaced.

[0024] In a possible implementation, determining the target pixel value corresponding to each processing area based on the pixel values ​​corresponding to each pixel point in each processing area includes:

[0025] Based on the pixel values ​​of each pixel point in each of the processing areas, the extreme pixel value corresponding to each of the processing areas is determined, and the extreme pixel value corresponding to each of the processing areas is used as the target pixel value corresponding to the processing area.

[0026] In this implementation, if the pixel extreme value is used as the target pixel value, the difference in pixel values ​​between the various processing areas after the pixel values ​​are replaced can be made more obvious.

[0027] In a possible implementation, when the video to be processed is a video shot in a road environment, the object type of the target object includes a face type and a license plate type.

[0028] In this implementation manner, both the target object of the face type and the target object of the license plate type are personal information. By blurring these two types of target objects, the security of the data can be effectively improved.

[0029] In one possible implementation, the identification of at least one type of target object in each frame of the video to be processed is performed using a pre-trained target neural network, which is a neural network trained using multiple sample images and capable of being used to identify multiple types of target objects.

[0030] In this implementation, since the target neural network trained using multiple sample images can recognize multiple types of target objects with high recognition accuracy, by using the trained target neural network to process each frame of the video to be processed, the various types of target objects included in each frame of the video can be accurately recognized, that is, various target objects related to personal information can be accurately recognized.

[0031] In a possible implementation, the target neural network includes a shared network and a plurality of branch networks, each of the branch networks is used to identify a type of target object;

[0032] Using a pre-trained target neural network, identifying at least one type of target object in each frame of the video to be processed, including:

[0033] Through the shared network in the target neural network, the video to be processed is subjected to video decoding processing to obtain each frame of video image corresponding to the video to be processed; and for each frame of the video image, the video image is subjected to multiple consecutive downsampling processing and upsampling processing;

[0034] At least one type of target object included in the video image is determined based on the results of the sampling process through multiple branch networks in the target neural network.

[0035] This implementation, based on continuous upsampling and downsampling processing, can fully extract feature information in the video image, and process the extracted information through multiple target branch networks, so as to accurately obtain various types of target objects in the video image.

[0036] In a possible implementation, for each frame of the video image, performing downsampling and upsampling processing on the video image for multiple consecutive times includes:

[0037] For each frame of the video image, the video image is continuously downsampled multiple times to obtain image feature information corresponding to each downsampling process; wherein the input information of a subsequent downsampling process in the multiple consecutive downsampling processes is the image feature information obtained by the previous downsampling process, and the input information of the first downsampling process is the video image;

[0038] The image feature information obtained by the last downsampling process is subjected to multiple consecutive upsampling processes to obtain initial category information corresponding to each upsampling process and initial detection frame information corresponding to the initial category information.

[0039] This implementation method can fully extract feature information in the video image by performing multiple downsampling and then multiple upsampling, thereby obtaining accurate initial category information and initial detection frame information.

[0040] In a possible implementation, determining at least one type of target object included in the video image based on the result after the sampling process through multiple branch networks in the target neural network includes:

[0041] Using a target branch network among the multiple branch networks that matches the initial category information, perform multiple feature extractions on the initial category information corresponding to each upsampling process to obtain target category information, and perform multiple feature extractions on the initial detection frame information corresponding to the upsampling process to obtain target detection frame information;

[0042] At least one type of target object included in the video image is determined based on the target category information and target detection frame information corresponding to each upsampling process.

[0043] In this embodiment, the target branch network that matches the initial category information is suitable for processing the initial category information. The target branch network that matches the initial category information is used to perform multiple consecutive feature extractions on the initial category information, which can achieve full information extraction, thereby obtaining accurate target category information, and the target branch network that matches the initial category information is used to perform multiple consecutive feature extractions on the initial detection frame information, which can accurately obtain target detection frame information of the target object that matches the target category information, where the detection frame information is used to reflect the position information of the target object; thereby, based on the target category information and target detection frame information corresponding to each upsampling process, each target object of each type included in the video image and its position information can be obtained.

[0044] In a possible implementation, determining at least one type of target object included in the video image based on the target category information and target detection frame information corresponding to each upsampling process obtained includes:

[0045] Determine the position information of each target object corresponding to each upsampling process based on the target category information and target detection frame information corresponding to each upsampling process;

[0046] Based on the position information, determining whether positions of multiple target objects corresponding to the multiple upsampling processes overlap;

[0047] In response to the overlap of the positions, confidences corresponding to the multiple target objects with overlapping positions are determined, and the target object with the highest confidence is used as the final target object;

[0048] The determined final target objects and target objects without position overlap are used as target objects in the video image.

[0049] Due to different upsampling processes, there is a possibility of identifying the corresponding initial detection frame information of the same target object. This implementation further processes the target category information and the target detection frame information obtained from each upsampling process, and it is inevitable that there will be multiple target objects with overlapping positions. The multiple target objects with overlapping positions here are likely to be the same target object. In addition, since the information obtained from each upsampling process is different, even if the same target object is obtained, there may be differences in confidence. Therefore, the target object with the highest confidence among the multiple target objects with overlapping positions is taken as the final target object, which can improve the accuracy of the target object finally identified.

[0050] In a second aspect, the present disclosure also provides a video processing device, including:

[0051] An acquisition module is used to acquire the video to be processed;

[0052] an identification module, used to identify at least one type of target object in each frame of the video to be processed, each type of target object being related to personal information;

[0053] A processing module, used for performing blur processing on the target object in the identified video image to obtain a target image;

[0054] The generating module is used to generate a blurred target video based on the target image corresponding to each frame of the video image.

[0055] In a possible implementation manner, the processing module is configured to extract an initial sub-image corresponding to each of the target objects from the video image in response to identifying a plurality of the target objects in the video image;

[0056] Performing fuzzy processing on each of the initial sub-images to obtain a target sub-image corresponding to each of the initial sub-images;

[0057] Each of the target sub-images is used to replace the corresponding initial sub-image in the video image to obtain the target image.

[0058] In a possible implementation manner, the processing module is used to perform blur processing on any of the initial sub-images according to the following steps:

[0059] Dividing the initial sub-image into a plurality of processing regions;

[0060] Determine a target pixel value corresponding to each processing area based on the pixel value of each pixel point in each processing area;

[0061] The pixel value corresponding to each pixel point in each processing area is replaced with the determined target pixel value to obtain the target sub-image corresponding to the initial sub-image.

[0062] In a possible implementation, the processing module is used to determine the average pixel value corresponding to each processing area based on the pixel values ​​of each pixel point in each processing area, and use the average pixel value corresponding to each processing area as the target pixel value corresponding to the processing area.

[0063] In a possible implementation, the processing module is used to determine the pixel value extreme value corresponding to each processing area based on the pixel values ​​of each pixel point in each processing area, and use the pixel value extreme value corresponding to each processing area as the target pixel value corresponding to the processing area.

[0064] In a possible implementation, when the video to be processed is a video shot in a road environment, the object type of the target object includes a face type and a license plate type.

[0065] In a possible implementation, the identification of at least one type of target object in each frame of the video to be processed is performed using a pre-trained target neural network, which is a neural network trained using multiple sample images and capable of being used to identify multiple types of target objects.

[0066] In a possible implementation, the target neural network includes a shared network and a plurality of branch networks, and each branch network is used to identify a type of target object;

[0067] The recognition module is used to recognize at least one type of target object in each frame of the video to be processed by using a pre-trained target neural network, including:

[0068] Through the shared network in the target neural network, the video to be processed is subjected to video decoding processing to obtain each frame of video image corresponding to the video to be processed; and for each frame of the video image, the video image is subjected to multiple consecutive downsampling processing and upsampling processing;

[0069] At least one type of target object included in the video image is determined based on the results of the sampling process through multiple branch networks in the target neural network.

[0070] In a possible implementation, the recognition module is used to perform multiple downsampling processes on the video image for each frame of the video image, and obtain image feature information corresponding to each downsampling process; wherein the input information of a subsequent downsampling process in the multiple consecutive downsampling processes is the image feature information obtained by the previous downsampling process, and the input information of the first downsampling process is the video image;

[0071] And, it is used to perform multiple consecutive upsampling processes on the image feature information obtained by the last downsampling process, and obtain initial category information corresponding to each upsampling process, and initial detection frame information corresponding to the initial category information.

[0072] In a possible implementation, the recognition module is configured to use a target branch network among the multiple branch networks that matches the initial category information to perform multiple feature extractions on the initial category information corresponding to each upsampling process to obtain target category information, and to perform multiple feature extractions on the initial detection frame information corresponding to the upsampling process to obtain target detection frame information;

[0073] At least one type of target object included in the video image is determined based on the target category information and target detection frame information corresponding to each upsampling process.

[0074] In a possible implementation, the recognition module is used to determine the position information of each target object corresponding to each upsampling process based on the target category information and the target detection frame information corresponding to each upsampling process;

[0075] Based on the position information, determining whether positions of multiple target objects corresponding to the multiple upsampling processes overlap;

[0076] In response to the overlap of the positions, confidences corresponding to the multiple target objects with overlapping positions are determined, and the target object with the highest confidence is used as the final target object;

[0077] The determined final target objects and target objects without position overlap are used as target objects in the video image.

[0078] In a third aspect, an optional implementation of the present disclosure further provides a computer device, a processor, and a memory, wherein the memory stores machine-readable instructions executable by the processor, and the processor is used to execute the machine-readable instructions stored in the memory, and when the machine-readable instructions are executed by the processor, the machine-readable instructions perform the steps of the above-mentioned first aspect, or any possible implementation of the first aspect.

[0079] In a fourth aspect, an optional implementation of the present disclosure further provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed, the steps of the above-mentioned first aspect, or any possible implementation of the first aspect are executed.

[0080] For a description of the effects of the above-mentioned video processing apparatus, computer device, and computer-readable storage medium, please refer to the description of the above-mentioned video processing method, which will not be repeated here.

[0081] In order to make the above-mentioned objectives, features and advantages of the present disclosure more obvious and easy to understand, preferred embodiments are specifically cited below and described in detail with reference to the attached drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0082] In order to more clearly illustrate the technical solutions of the embodiments of the present disclosure, the following is a brief introduction to the drawings required for use in the embodiments. The drawings herein are incorporated into the specification and constitute a part of the specification. These drawings illustrate embodiments consistent with the present disclosure and are used together with the specification to illustrate the technical solutions of the present disclosure. It should be understood that the following drawings only illustrate certain embodiments of the present disclosure and should not be regarded as limiting the scope. For ordinary technicians in this field, other relevant drawings can also be obtained based on these drawings without creative work.

[0083] Figure 1 A flow chart of a video processing method provided by an embodiment of the present disclosure is shown;

[0084] Figure 2 A schematic diagram of the structure of a shared network provided by an embodiment of the present disclosure is shown;

[0085] Figure 3 A schematic diagram of processing a frame of video image in a video to be processed provided by an embodiment of the present disclosure is shown;

[0086] Figure 4 A schematic diagram showing a comparison between an initial sub-image and a target sub-image provided by an embodiment of the present disclosure is shown;

[0087] Figure 5 A schematic diagram of a video processing device provided by an embodiment of the present disclosure is shown;

[0088] Figure 6 A schematic diagram of the structure of a computer device provided by an embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0089] In order to make the purpose, technical scheme and advantages of the embodiments of the present disclosure clearer, the technical scheme in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the drawings in the embodiments of the present disclosure. Obviously, the described embodiments are only part of the embodiments of the present disclosure, rather than all of the embodiments. The components of the embodiments of the present disclosure generally described and shown here can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present disclosure is not intended to limit the scope of the present disclosure claimed for protection, but merely represents the selected embodiments of the present disclosure. Based on the embodiments of the present disclosure, all other embodiments obtained by those skilled in the art without making creative work belong to the scope of protection of the present disclosure.

[0090] In addition, the terms "first", "second", etc. in the specification and claims of the embodiments of the present disclosure and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments described herein can be implemented in an order other than that illustrated or described herein. The "plurality or several" mentioned in this article refers to two or more. "And / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. The character " / " generally indicates that the objects associated before and after are in an "or" relationship.

[0091] According to research, in order to improve the security of video data, it is necessary to anonymize sensitive information in the video. However, in the prior art, the anonymization of sensitive information in the video is achieved by using a video tracking algorithm. Specifically, after determining the sensitive information in the first frame of the video, the video tracking algorithm is used to track and locate the sensitive information, and then anonymize it. However, due to the tracking accuracy of the video tracking algorithm, there may be failures in tracking sensitive information, or large tracking errors, resulting in poor anonymization of sensitive information and failure to ensure the security of video data.

[0092] Based on the above research, the present disclosure provides a video processing method, apparatus, computer equipment and storage medium, which can remove sensitive information in each frame of video image by identifying various types of target objects related to personal information in each frame of video image and then blurring the target objects in each frame of video image, thereby obtaining a target video with various sensitive information removed, that is, a desensitized target video, which effectively improves the security of video data.

[0093] The defects existing in the above solutions are the results obtained by the inventor after practice and careful research. Therefore, the discovery process of the above problems and the solutions proposed by the present disclosure for the above problems below should be the contributions made by the inventor to the present disclosure during the disclosure process.

[0094] It should be noted that similar reference numerals and letters denote similar items in the following drawings, and therefore, once an item is defined in one drawing, further definition and explanation thereof is not required in subsequent drawings.

[0095] To facilitate understanding of the present embodiment, a video processing method disclosed in the present embodiment is first introduced in detail. The executor of the video processing method provided in the present embodiment is generally a computer device with certain computing capabilities. In some possible implementation methods, the video processing method can be implemented by a processor calling computer-readable instructions stored in a memory.

[0096] The video processing method provided by the embodiment of the present disclosure is described below by taking a computer device as an example of an execution subject.

[0097] like Figure 1 As shown, it is a flowchart of a video processing method provided by an embodiment of the present disclosure, which may include the following steps:

[0098] S101: Obtain the video to be processed.

[0099] Here, the video to be processed may be a video including one or more types of target objects shot by any shooting device, and each type of target object may include one or more. For example, the video to be processed may be a video of the road environment ahead shot by a driving recorder, and personal information such as license plate information, driver or passenger face information, etc. may appear in the video to be processed.

[0100] Each type of target object is related to personal information, which is sensitive information in the image, such as face information, license plate information, etc. The target object can be an object related to personal information in the video, for example, the target object can be each face, each license plate number, each phone number, etc. that appears in the video.

[0101] S102: Identify at least one type of target object in each frame of the video to be processed; each type of target object is related to personal information.

[0102] In one embodiment, identifying at least one type of target object in each frame of the video to be processed can be performed using a pre-trained target neural network, and the target neural network can be a neural network trained using multiple sample images and capable of being used to identify multiple types of target objects.

[0103] The target neural network is a pre-trained neural network that can recognize multiple types of target objects in the video. Each type is an object type corresponding to a personal information, such as the object type corresponding to face information is face type, the object type corresponding to phone number information is phone number type, the object type corresponding to text information is text type, etc.

[0104] In the process of pre-training the target neural network, a large number of sample images in different scenes can be used, each sample image can include one or more types of sample objects, and the number of sample objects included in each sample image is different. Even if multiple sample images of the same type and the same number are included, the scenes corresponding to each sample image can be different. By using a large number of sample images to train the target neural network, a reliable recognition accuracy and recognition processing suitable for videos in various scenes and including various types of target objects can be obtained.

[0105] The type of target object that the trained target neural network can recognize can be set according to the actual application scenario, and this disclosure does not specifically limit it. In specific applications, it is only necessary to select sample images including any type of sample objects that need to be recognized, and train the target neural network to obtain a target neural network that can recognize any type of target object.

[0106] When this step is implemented, after the video to be processed is obtained, it can be input into a pre-trained target neural network, and the target neural network is used to first perform video decoding and frame cutting processing on it to obtain each frame of video image included in the video to be processed. Then, for each frame of video image, the frame of video image can be identified and processed to determine each target object of each type included in the frame of video image. For example, the convolution layer in the target neural network can be used to perform convolution processing on the video image continuously for multiple times to extract the image feature information of the video image, and then determine the feature information belonging to the object type and the feature information belonging to each target object based on the image feature information, and then determine the type of target object included in the video image and each type of target object.

[0107] In this way, based on the recognition processing of each frame of video image, various types of target objects included in each frame of video image can be accurately determined, that is, various types of target objects related to personal information included in the video to be processed can be accurately determined.

[0108] S103: performing blur processing on the target object in the identified video image to obtain a target image.

[0109] The target image is an image obtained by desensitizing various types of target objects included in the video image.

[0110] In a specific implementation, after identifying each type of target object included in each frame of video image based on the above S102, blur processing may be performed on each target object in each frame of video image.

[0111] The fuzzy processing method may include at least one of the following:

[0112] Method 1: Use the preset fill pattern to cover or replace the target object.

[0113] Method 2: Replace the pixel value of each pixel point corresponding to the target object with a preset pixel value.

[0114] Method three: replace the pixel values ​​of the pixels corresponding to some key areas of the target object (such as the facial features of a person, all or part of the license plate number) with the target pixel values.

[0115] Method 4: According to the type of target object and the association between the preset type and the fuzzy processing method (for example, the above three methods may be included), determine the target processing method corresponding to each type of target object identified, and then use the processing method to fuzzy the target object.

[0116] Method 5: cropping the image area corresponding to the target object from the video image, and then pasting an object schematic diagram (such as a contour diagram) of the same proportion and shape back to the image area corresponding to the target object.

[0117] Furthermore, after blurring each target object included in each frame of video image by using any of the above methods, a target image corresponding to each frame of video image can be obtained.

[0118] S104: Generate a blurred target video based on the target image corresponding to each frame of the video image.

[0119] After obtaining the target images corresponding to each frame of video images, the target images corresponding to each frame of video images can be merged according to the image sequence corresponding to each frame of video images in the video to be processed, so as to obtain the target video corresponding to the video to be processed and after blurring. The target video includes each target object after blurring.

[0120] In this way, by identifying various types of target objects related to personal information in each frame of video image, and then blurring the target objects in each frame of video image, it is possible to remove sensitive information in each frame of video image, and then obtain a target video with various sensitive information removed, that is, a desensitized target video, which effectively improves the security of video data.

[0121] In one embodiment, the target neural network includes a shared network and a plurality of branch networks, each branch network is used to identify a type of target object, and the step of using the pre-trained target neural network to identify at least one type of target object in each frame of the video to be processed can be implemented according to the following steps:

[0122] S102-1: Perform video decoding processing on the video to be processed through the shared network in the target neural network to obtain each frame of video image corresponding to the video to be processed; and for each frame of video image, perform downsampling processing and upsampling processing on the video image for multiple consecutive times.

[0123] Here, the shared network is a network that processes the video to be processed and each frame of the video included in the video, and is used to preliminarily extract the feature information in each frame of the video. After obtaining any video to be processed, it is necessary to first use the shared network to process it, and then use the branch network in the target neural network to further process the result of the shared network processing.

[0124] The shared network may include two parts, one part is a downsampling network and the other part is an upsampling network, wherein the downsampling network is used to downsample the video image, and the upsampling network is used to upsample the sampling result of the downsampling network. Both the upsampling network and the downsampling network may include multiple sampling layers, and the number of sampling layers of the upsampling network and the downsampling network is the same, and the depth, accuracy and completeness of the feature information collected by different sampling layers are different. In a specific implementation, the downsampling network may be a residual network RetNet, and the upsampling network may be a feature pyramid network Feature Pyramid Net. Alternatively, the downsampling network and the multiple sampling layers in the downsampling network may be multiple convolution layers with different convolution kernels.

[0125] Different branch networks are used to identify different types of target objects. One branch network can only identify one type of target object. The number of branch networks included in the target neural network can be determined according to the number of target object types that the target neural network needs to identify. For example, when the target neural network needs to identify two types of target objects, namely, face type and license plate type, the branch networks included in the target neural network may be a face recognition branch network and a license plate recognition branch network.

[0126] In specific implementation, after obtaining the video to be processed, the shared network can be used to perform video decoding processing on the video to be processed to obtain each frame of the video image corresponding to the video to be processed. Then, for each frame of the video image, multiple sampling layers in the downsampling network can be used to perform multiple consecutive downsampling processing on it, and then multiple sampling layers in the upsampling network can be used to perform multiple consecutive upsampling processing on the result of the last downsampling to obtain the sampling processing result corresponding to the video image.

[0127] S102-2: Determine at least one type of target object included in the video image based on the results of sampling processing through multiple branch networks in the target neural network.

[0128] In this step, each branch network can be used to process the sampling results respectively to obtain the output of each branch grid, and the output of each branch network is used as each target object included in the video image.

[0129] Here, since different branch networks can output target objects of different object types, if the video image does not include information related to the object type that a branch network can identify, when the branch network is used to process the above-mentioned sampling result corresponding to the video image, the information may not be output, or an error message may be output. Therefore, the target object in the video image is determined directly using the output of each branch network, and the target object finally obtained may include at least one type.

[0130] Alternatively, for S102-2, after obtaining the result after sampling processing, at least one target branch network for processing the result can be selected from multiple branch networks based on the result after sampling processing, and then the result after sampling processing can be processed using the determined target branch network to determine at least one type of target object included in the video image.

[0131] During specific implementation, the target branch network can be determined in the following manner:

[0132] Method 1: Based on the processed result, each object type corresponding to each target object included in the video image can be determined, and the branch network corresponding to each object type can be selected as each target branch network.

[0133] Method 2: In the process of executing the above S102-1, for each frame of video image, multiple sampling layers in the downsampling network can be used to perform continuous downsampling processing on the video image multiple times to obtain a first result of the video image corresponding to the downsampling processing, and multiple sampling layers in the upsampling network can be used to perform continuous upsampling processing on the video image multiple times to obtain a second result of the video image corresponding to the upsampling processing. Then, based on the first result, at least one target branch network can be selected from the multiple branch networks, and based on the second result, at least one target branch network can be selected from the multiple branch networks, and then, the target branch network corresponding to the first result and the target branch network corresponding to the second result are used as the final target branch networks.

[0134] Method three: in the process of executing the above S102-1, for each frame of video image, after using the downsampling network to process and obtain the above first result, the first result and the video image are merged and input into the upsampling network, and the upsampling network is used to perform multiple upsampling processes on the merged first result and the video image to obtain the final sampling result; finally, the sampling result is used to select at least one target branch network from multiple branch networks.

[0135] After determining each target branch network, the results after sampling processing can be input into each target branch network respectively, and each target branch network can be used to process the results after sampling processing respectively to obtain each target branch network output that matches the recognition type corresponding to the branch network, thereby obtaining each target object of each type included in the video image.

[0136] Alternatively, after determining each target branch network, for each target branch network, the result related to the recognition type corresponding to the target branch network in the sampling processing result can be input into the target branch network and processed to obtain each target object output by the target branch network and matching the recognition type corresponding to the target branch network. Finally, based on each target object of each type output by each target branch network, each target object of each type included in the video image is determined.

[0137] In one embodiment, the step of performing multiple consecutive downsampling and upsampling processes on each frame of the video image in S102-1 may be implemented according to the following steps:

[0138] S102-1-1: For each frame of video image, perform downsampling processing on the video image for multiple times continuously to obtain image feature information corresponding to each downsampling processing.

[0139] S102-1-2: Perform multiple consecutive upsampling processes on the image feature information obtained from the last downsampling process, and obtain initial category information corresponding to each upsampling process, and initial detection frame information corresponding to the initial category information.

[0140] The input information of the next downsampling process in the multiple consecutive downsampling processes is the image feature information obtained in the previous downsampling process, and the input information of the first downsampling process is a video image. The total number of the multiple consecutive upsampling processes is the same as the total number of the multiple consecutive downsampling processes, and the input information of the next upsampling process in the multiple consecutive upsampling processes is the initial category information, initial detection frame information, and image feature information obtained in the previous upsampling process, and the downsampling process matching the upsampling process, wherein the downsampling process matches the upsampling process means that the sum of the sampling sequence bits corresponding to the upsampling process and the sampling sequence bits corresponding to the downsampling process is equal to the total number plus 1.

[0141] The result after the sampling process in S102 - 1 is the initial category information corresponding to each up-sampling process, and the initial detection frame information corresponding to the initial category information.

[0142] The initial category information is used to characterize the object type of the target object in the video image recognized by the shared network. The initial category information corresponding to each upsampling process can be the same or different; the initial detection frame information is used to reflect the initial predicted position information of each target object corresponding to the initial category information. For example, the initial category information may include two face-type target objects and one license plate-type target object, and the initial detection frame information may include initial detection frames corresponding to the two face-type target objects and an initial detection frame corresponding to the license plate-type target object.

[0143] like Figure 2 FIG. 1 is a schematic diagram of a shared network structure provided by an embodiment of the present disclosure. Figure 2 The downsampling network in the shared network shown includes three downsampling layers: a first downsampling layer corresponding to the first sampling order bit, a second downsampling layer corresponding to the second sampling order bit, and a third downsampling layer corresponding to the third sampling order bit; the upsampling network also includes three upsampling layers: a first upsampling layer corresponding to the first sampling order bit, a second upsampling layer corresponding to the second sampling order bit, and a third upsampling layer corresponding to the third sampling order bit. Figure 2 As just an example, the number of sampling layers included in the up- and down-sampling networks can be set according to actual sampling needs and is not limited here.

[0144] Below Figure 2The downsampling network shown is a detailed description of the above S102-1-1 and S102-1-2:

[0145] In a specific implementation, for each frame of video image, the frame of video image can be first input into the first downsampling layer in the downsampling network, and the downsampling layer is used to downsample the frame of video image to extract image feature information corresponding to the first downsampling layer, such as color feature information, texture feature information, etc.

[0146] Then, the image feature information extracted by the first downsampling layer is input to the second downsampling layer, and the image feature information is further downsampled by the second downsampling layer to extract the image feature information corresponding to the second downsampling layer of the video image.

[0147] Then, the image feature information extracted by the second downsampling layer is input to the third downsampling layer, and the image feature information is further downsampled by the third downsampling layer to extract the image feature information corresponding to the third downsampling layer of the video image. That is, the video image is downsampled three times in a row, and the total number of downsampling processes is 3 times.

[0148] Furthermore, the image feature information obtained from the last downsampling process, that is, the image feature information of the third downsampling layer corresponding to the third sampling order position of the extracted video image, can be input into the first upsampling layer located at the first sampling order position in the upsampling network, and the image feature information is upsampled using the first upsampling layer to obtain the initial category information corresponding to the first upsampling process and the initial detection box information corresponding to the initial category information.

[0149] Then, the initial category information corresponding to the first upsampling process, the initial detection frame information corresponding to the initial category information, and the image feature information obtained by the downsampling process matching the second upsampling process at the second sampling order position (that is, the image feature information obtained by sampling the second downsampling layer at the second sampling order position) are fused, and the fused results are input together to the second upsampling layer at the second sampling order position, and the second upsampling layer is used to upsample the fused results to obtain the initial category information corresponding to the second upsampling process and the initial detection frame information corresponding to the initial category information.

[0150] Then, the initial category information corresponding to the second upsampling process, the initial detection frame information corresponding to the initial category information, and the image feature information obtained by the downsampling process matching the third upsampling process at the third sampling order (i.e., the image feature information obtained by sampling the first downsampling layer at the first sampling order) are fused, and the fused results are input together to the third upsampling layer at the third sampling order, and the fused results are upsampled by the third upsampling layer to obtain the initial category information corresponding to the third upsampling process and the initial detection frame information corresponding to the initial category information. Here, the total number of upsampling is also 3 times.

[0151] Furthermore, for the above step of selecting the target branch network, after obtaining the initial category information corresponding to each upsampling process, the various object types corresponding to the initial category information can be determined according to the initial category information, and based on the various object types, a target branch network capable of identifying each object type can be selected from multiple branch networks. For example, when the object types included in the initial category information corresponding to an upsampling process are face type and vehicle type, the determined target branch network can be a face recognition branch network and a license plate recognition branch network; when the object type included in the initial category information corresponding to an upsampling process is only a face type, the determined target branch network can be a face recognition branch network.

[0152] Furthermore, based on the initial category information corresponding to each upsampling process, a target branch network corresponding to each upsampling process can be determined.

[0153] In one embodiment, the above S102-2 may be implemented according to the following steps:

[0154] S102-2-1: Utilizing a target branch network among multiple branch networks that matches the initial category information, perform multiple consecutive feature extractions on the initial category information corresponding to each upsampling process to obtain target category information, and perform multiple consecutive feature extractions on the initial detection frame information corresponding to the upsampling process to obtain target detection frame information.

[0155] Here, the target branch networks matching the initial category information are the target branch networks determined by using the method provided in the above embodiment.

[0156] The target category information is the detailed information of the target object in the video image that is output by the target branch network and is used to characterize the object type that can be recognized by the target branch network, and can characterize whether there is a target object in the video image that matches the object type that can be recognized by the target branch network. For example, when the target branch network is a face recognition branch network, the target category information can be the detailed information of each face output by the face recognition branch network, such as the specific pixel position of the face outline.

[0157] The target detection frame information is the final detection frame of each target object corresponding to the target category information.

[0158] Each branch network can be divided into two parts, one is a category information extraction network for processing the initial category information, and the other is a detection box information extraction network for processing the initial detection box information. Both networks include multiple convolutional layers, and the number of convolutional layers included in the two networks is the same.

[0159] In specific implementation, for the initial category information and initial detection frame information obtained in each upsampling process, the initial category information and initial detection frame information can be processed respectively using the respective target branch networks corresponding to the upsampling process. For example, in the case where the target branch network corresponding to the upsampling process includes two face recognition branch networks and a license plate recognition branch network, the initial category information and initial detection frame information corresponding to the upsampling process can be input into the face recognition branch network and the license plate recognition branch network; the initial category information and initial detection frame information are processed using the face recognition branch network to obtain the target category information and target detection frame information corresponding to the face recognition branch network; and the initial category information and initial detection frame information are processed using the license plate recognition branch network to obtain the target category information and target detection frame information corresponding to the license plate recognition branch network.

[0160] In this way, for the initial category information and initial detection frame information obtained in each upsampling process, each target branch network corresponding to the upsampling process can be used to determine the target category information and target detection frame information corresponding to the upsampling process.

[0161] The process of using each target branch network to determine the target category information and target detection box information can be specifically as follows:

[0162] For the input initial category information, the first convolution layer of the multiple convolution layers in the category information extraction network in the target branch network can be used to extract features from the input initial category information, and the result of the first extraction is input to the second convolution layer to obtain the result of the second extraction, and then the result of the second extraction is input to the next convolution layer to obtain the result of a new extraction, and so on, until the result of the last convolution layer extraction is obtained, and the result is used as the obtained target category information. Similarly, for the input initial detection frame information, the first convolution layer of the multiple convolution layers in the detection frame information extraction network in the target branch network can be used to extract features from the input initial detection frame information, and the result of the first extraction is input to the second convolution layer to obtain the result of the second extraction, and then the result of the second extraction is input to the next convolution layer to obtain the result of a new extraction, and so on, until the result of the last convolution layer extraction is obtained, and the result is used as the obtained target detection frame information.

[0163] S102-2-2: Determine at least one type of target object included in the video image based on the target category information and target detection frame information corresponding to each upsampling process.

[0164] In a specific implementation, the target category information and target detection frame information corresponding to each upsampling process can be merged to obtain the final target category information and target detection frame information corresponding to the video image, and the object type corresponding to the final target category information is used as the object type of the target object included in the video image, and the various objects in the detection frame corresponding to the final target detection frame information are used as the various target objects included in the video image.

[0165] In one embodiment, S102-2-2 may be implemented according to the following steps:

[0166] S102-2-2-1: Based on the target category information and target detection frame information corresponding to each upsampling process, determine the position information of each target object corresponding to each upsampling process.

[0167] Here, the target detection frame information is the detection frame, which can be a rectangular detection frame. The position information of the target object can include the upper left vertex of the detection frame corresponding to the target object, the corresponding pixel position information in the video image, and the length and width information of the detection frame, or can also be the pixel position information corresponding to each vertex of the detection frame corresponding to the target object in the video image.

[0168] During specific implementation, for the target category information and target detection frame information corresponding to each upsampling process, the number of object types included in the target category information corresponding to the upsampling process can be used to determine the number of target objects corresponding to the upsampling process, and then the detection frame of each target object corresponding to the upsampling process can be determined based on the detail information included in the target category information. Furthermore, the pixel position information and length and width information corresponding to the vertices of the detection frame of each target object corresponding to the upsampling process can be used as the position information of each target object corresponding to the upsampling process.

[0169] S102-2-2-2: Based on the position information, determine whether the positions of multiple target objects corresponding to the multiple upsampling processes overlap.

[0170] Here, the position information determined by S102-2-2-1 above includes the position information of each target object corresponding to each upsampling process, and since the target category information and target detection frame information corresponding to each upsampling process may partially overlap, that is, the target category information and target detection frame information corresponding to each upsampling process may correspond to the same target object. Therefore, after obtaining the position information of each target object, it is also possible to determine whether the positions of multiple target objects corresponding to multiple upsampling processes overlap based on the positions corresponding to each position information.

[0171] S102-2-2-3: In response to the overlap in positions, determine the confidences corresponding to the multiple target objects with overlapping positions, and use the target object with the highest confidence as the final target object.

[0172] In a specific implementation, based on each position information, multiple target objects with overlapping positions are found, and for each target object in the multiple target objects with overlapping positions, the confidence corresponding to each target object can be determined first. The confidence can be the confidence of the target category information corresponding to the target object, or the confidence of the target detection box information corresponding to the target object; or, it can also be the average of the confidences corresponding to the above two confidences.

[0173] Finally, according to the confidence corresponding to each target object among the multiple target objects with overlapping positions, the target object with the highest confidence can be used as the final target object at the overlapping position, and the other target objects can be deleted.

[0174] S102-2-2-4: The determined final target objects and target objects without position overlap are regarded as at least one type of target objects included in the video image.

[0175] In a specific implementation, the determined final target objects and the target objects without position overlap among the target objects corresponding to each upsampling process can be directly used as at least one type of target object included in the video image.

[0176] like Figure 3 FIG. 1 is a schematic diagram of processing a frame of a video image in a video to be processed provided by an embodiment of the present disclosure, wherein: Figure 3 There are two branch networks in it - face recognition branch network and vehicle recognition branch network. The two branch networks have the same number of convolutional layers. Figure 3 The ×4 in means that after 4 convolutional layers (each is an example), after using the shared network in the target neural network to output the initial category information and initial detection box information corresponding to each upsampling process, it can be input into different branch networks, such as Figure 3 As shown, the initial category information 1 and initial detection frame information 1 corresponding to the first upsampling process can be input into the face recognition branch network, the initial category information 2 and initial detection frame information 2 corresponding to the second upsampling process can be input into the license plate recognition branch network, and the initial category information 3 and initial detection frame information 3 corresponding to the third upsampling process can be input into the face recognition branch network and the license plate recognition branch network; then, the category information extraction network and the detection frame information extraction network in each recognition branch network are used to process the input initial category information and initial detection frame information respectively. Here, since the initial category information and initial detection frame information input into the face recognition branch network are the results of the first upsampling process and the third upsampling process, Figure 3 The target category information output by the face recognition branch network in includes target category information 1 corresponding to the initial category information 1 in the first upsampling process and target category information 3 corresponding to the initial category information 3 in the third upsampling process. Similarly, Figure 3 The target detection frame information output by the face recognition branch network also includes target detection frame information 1 corresponding to the first upsampling process and target detection frame information 3 corresponding to the third upsampling process. Figure 3 The license plate recognition branch network in only outputs the target category information 2 and target detection frame information 2 corresponding to the second upsampling process. Finally, at least one type of target object included in the video image can be determined based on the output target category information 1, 2, 3 and target detection frame information 1, 2, 3.

[0177] In one embodiment, the above-mentioned target neural network mentioned in the video processing method provided in the embodiment of the present disclosure may also include multiple target neural networks, each of which is used to identify a type of target object in the video to be processed. For example, the face neural network is used to identify the face in each frame of the video image, and the license plate neural network is used to identify the license plate in each frame of the video image. In specific implementation, the video to be processed can be input into the target neural network corresponding to the object type according to the object type of the target object that needs to be blurred, and then the target object identified by the target neural network is blurred to obtain the blurred target video. The specific recognition process of the target neural network used to identify each type of target object can refer to the recognition process of the corresponding branch network in the above embodiments, which will not be repeated here.

[0178] In one embodiment, S103 may be implemented according to the following steps:

[0179] S103 - 1 : In response to identifying a plurality of target objects in the video image, extracting an initial sub-image corresponding to each target object from the video image.

[0180] In a specific implementation, for each frame of video image, when it is determined that multiple target objects are identified from the frame of video image, an object identifier (Identity document, ID) can be assigned to each target object in response to identifying multiple target objects in the video image; then, based on the position information corresponding to each target object, the initial sub-image corresponding to each target object is deducted from the video image, and the object identifier corresponding to each target object is used as the image identifier of its corresponding initial sub-image.

[0181] In one embodiment, for the step of extracting the initial sub-image corresponding to each target object from the video image according to the position information corresponding to each target object, the image area corresponding to the detection frame of the target object can be directly extracted as the initial sub-image of the target object; or, the detection frame of the target object can be scaled according to a preset scaling ratio, and the image area corresponding to the scaled detection frame can be extracted as the initial sub-image of the target object.

[0182] In one implementation, even if only one target object is identified from the video image, the initial sub-image corresponding to the target object may be deducted from the video image based on the position information corresponding to the target object.

[0183] S103-2: Perform fuzzy processing on each initial sub-image to obtain a target sub-image corresponding to each initial sub-image.

[0184] In specific implementation, each initial sub-image deducted from the video image may be fuzzy processed to obtain a target sub-image corresponding to each initial sub-image, and the image identifier corresponding to each initial sub-image is used as the image identifier corresponding to the initial sub-image.

[0185] S103-3: Using each target sub-image to replace the corresponding initial sub-image in the video image to obtain a target image.

[0186] In a specific implementation, the target object corresponding to the target sub-image can be determined according to the image identifier corresponding to each target sub-image and the object identifier corresponding to each target object, and then the target sub-image can be used to replace the initial sub-image corresponding to the target object in the video image. Furthermore, based on the image identifier of the initial sub-image corresponding to each target sub-image and the object identifier corresponding to the target object, the initial sub-image corresponding to each target object in the video image can be replaced to obtain the target image.

[0187] In one embodiment, for any initial sub-image, blur processing is performed on the initial sub-image according to the following steps:

[0188] Step 1: Divide the initial sub-image into multiple processing areas.

[0189] In this step, the initial sub-image can be divided into a plurality of processing areas of equal size according to the image size corresponding to the initial sub-image. For example, when the image size corresponding to the initial sub-image is 100*80, the initial sub-image can be divided into 100 processing areas, and the area size of each processing area is 10*8. Alternatively, the initial sub-image can be divided into the number of processing areas according to the preset number of processing areas. Alternatively, the division method can be determined according to the object type of the target object. For example, when the object type of the target object is a face type, the initial sub-image can be divided into a plurality of processing areas according to the position information of the facial features; when the object type of the target object is a license plate type, the initial sub-image can be divided into a plurality of processing areas of equal size.

[0190] Step 2: Based on the pixel values ​​of each pixel point in each processing area, determine the target pixel value corresponding to each processing area.

[0191] In one embodiment, for each processing area obtained by division, a pixel mean corresponding to the processing area can be determined based on the pixel values ​​corresponding to each pixel point in the processing area, and the determined pixel mean is used as the target pixel value of the processing area.

[0192] In another embodiment, for each processing area obtained by division, the pixel extreme value corresponding to the processing area can be determined based on the pixel values ​​corresponding to each pixel point in the processing area, and the determined pixel extreme value can be used as the target pixel value of the processing area.

[0193] Alternatively, for each divided processing area, a pixel value may be randomly selected from the pixel values ​​corresponding to each pixel point in the processing area as the target pixel value.

[0194] Step 3: Replace the pixel value corresponding to each pixel point in each processing area with the determined target pixel value to obtain the target sub-image corresponding to the initial sub-image.

[0195] In specific implementation, for each processing area, the pixel values ​​corresponding to each pixel point in the processing area can be replaced with the target pixel values ​​corresponding to the processing area. After completing the pixel value replacement of each processing area, the target sub-image is obtained. Figure 4 , which is a schematic diagram of comparison between an initial sub-image and a target sub-image provided by an embodiment of the present disclosure.

[0196] In one embodiment, when the video to be processed is a video shot in a road environment, since the target objects appearing in the video shot in the road environment are mostly faces of pedestrians or license plates of other vehicles, and faces and license plates are sensitive information that need to be desensitized, therefore, for the video shot in the road environment, the object types of the target objects include face type and license plate type.

[0197] Exemplarily, the video to be processed may be a driving video uploaded by a user and shot by a driving recorder, and the object type of the target object may include: a face type and a license plate type.

[0198] Of course, in addition to the above object types, it may also include object types corresponding to various target objects that may appear in the video, such as target animal type, target building type, target object type, etc. Regarding the specific object type, the embodiment of the present disclosure does not specifically limit it.

[0199] In addition, an embodiment of the present disclosure also provides a method for training a target neural network using multiple sample images. Specifically, it is necessary to first obtain multiple sample images, wherein each sample image includes at least one type of sample object, and the sample objects and the number of sample objects included in different sample images may be different.

[0200] Then, the sample image can be input into the target neural network to be trained, and the sample image can be recognized and processed using the shared network in the target neural network to be trained to determine the initial predicted category information of the sample image corresponding to each upsampling process, as well as the initial predicted detection box information corresponding to the initial predicted category information.

[0201] Afterwards, for the initial prediction category information corresponding to each upsampling process, at least one target branch network matching the initial prediction category information can be screened out from multiple branch networks, and the initial prediction category information and the initial prediction detection frame information corresponding to the initial prediction category information can be processed using the screened target branch network to determine the target prediction category information and target prediction detection frame information corresponding to the upsampling process. Furthermore, the predicted sample object included in the sample image can be determined based on the target prediction category information and target prediction detection frame information corresponding to each upsampling process.

[0202] Finally, the first loss value corresponding to the shared network can be determined based on the initial prediction category information and initial prediction detection box information output by the shared network, as well as the standard initial category information and standard initial detection box information; the second loss value corresponding to each branch network can be determined based on the target prediction category information and target prediction detection box information output by each branch network when it is a target branch network, as well as the standard target prediction category information and standard prediction detection box information; thereafter, the shared network is iteratively trained using the determined first loss value, and the branch network is iteratively trained using the second loss value corresponding to each branch network, so as to adjust the network parameter values ​​of the shared network and each branch network until the preset training cutoff condition is met, thereby obtaining the trained shared network and each branch network. The preset training cutoff condition may include that the number of iterative training rounds reaches the preset number of rounds and / or the prediction accuracy of the trained network reaches the preset accuracy.

[0203] Furthermore, a third loss value may be determined based on the standard sample object corresponding to each sample image and the predicted sample object output by the target neural network, and the target neural network to be trained may be iteratively trained using the third loss value. In addition, in addition to determining the second loss value corresponding to each branch network, a fourth loss value corresponding to each branch network may be determined based on each predicted object output by each branch network and the corresponding standard object, and each branch network may be iteratively trained using the fourth loss value.

[0204] Finally, after ensuring that the shared network and each branch network have been trained, a trained target neural network is obtained.

[0205] Those skilled in the art will appreciate that, in the above method of specific implementation, the order in which the steps are written does not imply a strict execution order and does not constitute any limitation on the implementation process. The specific execution order of the steps should be determined by their functions and possible internal logic.

[0206] Based on the same inventive concept, a video processing device corresponding to the video processing method is also provided in the embodiment of the present disclosure. Since the principle of solving the problem by the device in the embodiment of the present disclosure is similar to the above-mentioned video processing method in the embodiment of the present disclosure, the implementation of the device can refer to the implementation of the method, and the repeated parts will not be repeated.

[0207] like Figure 5 FIG. 1 is a schematic diagram of a video processing device provided by an embodiment of the present disclosure, comprising:

[0208] An acquisition module 501 is used to acquire a video to be processed;

[0209] An identification module 502, configured to identify at least one type of target object in each frame of the video to be processed, wherein each type of target object is related to personal information;

[0210] The processing module 503 is used to perform blur processing on the identified target object in the video image to obtain a target image;

[0211] The generating module 504 is used to generate a blurred target video based on the target image corresponding to each frame of the video image.

[0212] In a possible implementation, the processing module 503 is configured to extract an initial sub-image corresponding to each of the target objects from the video image in response to identifying a plurality of the target objects in the video image;

[0213] Performing fuzzy processing on each of the initial sub-images to obtain a target sub-image corresponding to each of the initial sub-images;

[0214] Each of the target sub-images is used to replace the corresponding initial sub-image in the video image to obtain the target image.

[0215] In a possible implementation manner, the processing module 503 is configured to perform blur processing on any of the initial sub-images according to the following steps:

[0216] Dividing the initial sub-image into a plurality of processing regions;

[0217] Determine a target pixel value corresponding to each processing area based on the pixel value of each pixel point in each processing area;

[0218] The pixel value corresponding to each pixel point in each processing area is replaced with the determined target pixel value to obtain the target sub-image corresponding to the initial sub-image.

[0219] In a possible implementation, the processing module 503 is used to determine the mean pixel value corresponding to each processing area based on the pixel values ​​of each pixel point in each processing area, and use the mean pixel value corresponding to each processing area as the target pixel value corresponding to the processing area.

[0220] In a possible implementation, the processing module 503 is used to determine the pixel value extreme value corresponding to each processing area based on the pixel values ​​of each pixel point in each processing area, and use the pixel value extreme value corresponding to each processing area as the target pixel value corresponding to the processing area.

[0221] In a possible implementation, when the video to be processed is a video shot in a road environment, the object type of the target object includes a face type and a license plate type.

[0222] In a possible implementation, the identification of at least one type of target object in each frame of the video to be processed is performed using a pre-trained target neural network, which is a neural network trained using multiple sample images and capable of being used to identify multiple types of target objects.

[0223] In a possible implementation, the target neural network includes a shared network and a plurality of branch networks, and each branch network is used to identify a type of target object;

[0224] The recognition module 502 is used to recognize at least one type of target object in each frame of the video to be processed by using a pre-trained target neural network, including:

[0225] Through the shared network in the target neural network, the video to be processed is subjected to video decoding processing to obtain each frame of video image corresponding to the video to be processed; and for each frame of the video image, the video image is subjected to multiple consecutive downsampling processing and upsampling processing;

[0226] At least one type of target object included in the video image is determined based on the results of the sampling process through multiple branch networks in the target neural network.

[0227] In a possible implementation, the recognition module 502 is used to perform multiple downsampling processes on the video image for each frame of the video image, and obtain image feature information corresponding to each downsampling process; wherein the input information of a subsequent downsampling process in the multiple consecutive downsampling processes is the image feature information obtained by the previous downsampling process, and the input information of the first downsampling process is the video image;

[0228] And, it is used to perform multiple consecutive upsampling processes on the image feature information obtained by the last downsampling process, and obtain initial category information corresponding to each upsampling process, and initial detection frame information corresponding to the initial category information.

[0229] In a possible implementation, the recognition module 502 is configured to use a target branch network among the multiple branch networks that matches the initial category information to perform multiple feature extractions on the initial category information corresponding to each upsampling process to obtain target category information, and to perform multiple feature extractions on the initial detection frame information corresponding to the upsampling process to obtain target detection frame information;

[0230] At least one type of target object included in the video image is determined based on the target category information and target detection frame information corresponding to each upsampling process.

[0231] In a possible implementation, the identification module 502 is used to determine the position information of each target object corresponding to each upsampling process based on the target category information and the target detection frame information corresponding to each upsampling process;

[0232] Based on the position information, determining whether positions of multiple target objects corresponding to the multiple upsampling processes overlap;

[0233] In response to the overlap of the positions, confidences corresponding to the multiple target objects with overlapping positions are determined, and the target object with the highest confidence is used as the final target object;

[0234] The determined final target objects and target objects without position overlap are used as target objects in the video image.

[0235] For descriptions of the processing flow of each module in the device and the interaction flow between each module, reference may be made to the relevant descriptions in the above method embodiment, which will not be described in detail here.

[0236] The present disclosure also provides a computer device, such as Figure 6 FIG. 1 is a schematic diagram of a computer device structure provided by an embodiment of the present disclosure, including:

[0237] A processor 61 and a memory 62; the memory 62 stores machine-readable instructions executable by the processor 61, and the processor 61 is used to execute the machine-readable instructions stored in the memory 62. When the machine-readable instructions are executed by the processor 61, the processor 61 performs the following steps: S101: obtaining a video to be processed; S102: identifying at least one type of target object in each frame of the video to be processed; each type of target object is related to personal information; S103: blurring the target object in the identified video image to obtain a target image; and S104: generating a blurred target video based on the target images corresponding to each frame of the video image.

[0238] The above-mentioned memory 62 includes internal memory 621 and external memory 622; the memory 621 here is also called internal memory, which is used to temporarily store the calculation data in the processor 61, as well as the data exchanged with the external memory 622 such as the hard disk. The processor 61 exchanges data with the external memory 622 through the internal memory 621.

[0239] The specific execution process of the above instructions can refer to the steps of the video processing method described in the embodiment of the present disclosure, which will not be repeated here.

[0240] The present disclosure also provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the steps of the video processing method described in the above method embodiment are executed. The storage medium can be a volatile or non-volatile computer-readable storage medium.

[0241] The computer program product of the video processing method provided in the embodiments of the present disclosure includes a computer-readable storage medium storing program code, and the instructions included in the program code can be used to execute the steps of the video processing method described in the above method embodiments. For details, please refer to the above method embodiments, which will not be repeated here.

[0242] The computer program product may be implemented in hardware, software or a combination thereof. In one optional embodiment, the computer program product is embodied as a computer storage medium, and in another optional embodiment, the computer program product is embodied as a software product, such as a software development kit (SDK) and the like.

[0243] Those skilled in the art can clearly understand that, for the convenience and simplicity of description, the specific working process of the device described above can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here. In the several embodiments provided in the present disclosure, it should be understood that the disclosed device and method can be implemented in other ways. The device embodiments described above are merely schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some communication interfaces, and the indirect coupling or communication connection of the device or unit can be electrical, mechanical or other forms.

[0244] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0245] In addition, each functional unit in each embodiment of the present disclosure may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0246] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a non-volatile computer-readable storage medium that is executable by a processor. Based on this understanding, the technical solution of the present disclosure, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present disclosure. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0247] Finally, it should be noted that the above-described embodiments are only specific implementation methods of the present disclosure, which are used to illustrate the technical solutions of the present disclosure, rather than to limit them. The protection scope of the present disclosure is not limited thereto. Although the present disclosure is described in detail with reference to the above-described embodiments, ordinary technicians in the field should understand that any technician familiar with the technical field can still modify the technical solutions recorded in the above-described embodiments within the technical scope disclosed in the present disclosure, or can easily think of changes, or make equivalent replacements for some of the technical features therein; and these modifications, changes or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present disclosure, and should be included in the protection scope of the present disclosure. Therefore, the protection scope of the present disclosure should be based on the protection scope of the claims.

Claims

1. A video processing method, characterized in that: include: Get the video to be processed; Using a pre-trained target neural network, identifying at least one type of target object in each frame of the video to be processed, each type of target object being related to personal information; the target neural network includes a shared network and a plurality of branch networks, each of the branch networks being used to identify a type of target object; Using a pre-trained target neural network, identifying at least one type of target object in each frame of the video to be processed, including: Through the shared network, the video to be processed is subjected to video decoding processing to obtain each frame of video image corresponding to the video to be processed; and for each frame of the video image, the video image is subjected to multiple consecutive downsampling processing to obtain image feature information corresponding to each downsampling processing; wherein the input information of the subsequent downsampling processing in the multiple consecutive downsampling processing is the image feature information obtained by the previous downsampling processing, wherein the input information of the first downsampling processing is the video image; Performing multiple upsampling processes on the image feature information obtained by the last downsampling process, and obtaining initial category information corresponding to each upsampling process, and initial detection frame information corresponding to the initial category information; Using a target branch network among the multiple branch networks that matches the initial category information, perform multiple feature extractions on the initial category information corresponding to each upsampling process to obtain target category information, and perform multiple feature extractions on the initial detection frame information corresponding to the upsampling process to obtain target detection frame information; Determine at least one type of target object included in the video image based on the target category information and target detection frame information corresponding to each upsampling process; Performing blur processing on the identified target object in the video image to obtain a target image; Based on the target images corresponding to each frame of the video image, a blurred target video is generated.

2. The method according to claim 1, characterized in that The step of performing blur processing on the identified target object in the video image to obtain the target image includes: In response to identifying a plurality of the target objects in the video image, extracting an initial sub-image corresponding to each of the target objects from the video image; Performing fuzzy processing on each of the initial sub-images to obtain a target sub-image corresponding to each of the initial sub-images; Each of the target sub-images is used to replace the corresponding initial sub-image in the video image to obtain the target image.

3. The method according to claim 2, characterized in that For any of the initial sub-images, blur processing is performed on the initial sub-image according to the following steps: Dividing the initial sub-image into a plurality of processing regions; Determine a target pixel value corresponding to each processing area based on the pixel value of each pixel point in each processing area; The pixel value corresponding to each pixel point in each processing area is replaced with the determined target pixel value to obtain the target sub-image corresponding to the initial sub-image.

4. The method according to claim 3, characterized in that The determining of the target pixel value corresponding to each processing area based on the pixel values ​​corresponding to each pixel point in each processing area includes: Based on the pixel values ​​of each pixel point in each of the processing areas, the average pixel value corresponding to each of the processing areas is determined, and the average pixel value corresponding to each of the processing areas is used as the target pixel value corresponding to the processing area.

5. The method according to claim 4, characterized in that The determining of the target pixel value corresponding to each processing area based on the pixel values ​​corresponding to each pixel point in each processing area includes: Based on the pixel values ​​of each pixel point in each of the processing areas, the extreme pixel value corresponding to each of the processing areas is determined, and the extreme pixel value corresponding to each of the processing areas is used as the target pixel value corresponding to the processing area.

6. The method according to any one of claims 1 to 5, characterized in that: In the case that the video to be processed is a video shot in a road environment, the object type of the target object includes a face type and a license plate type.

7. The method according to any one of claims 1 to 6, characterized in that: The target neural network is a neural network trained using a plurality of sample images and can be used to identify various types of target objects.

8. The method according to claim 1, characterized in that The determining, based on the target category information and target detection frame information corresponding to each upsampling process, at least one type of target object included in the video image comprises: Determine the position information of each target object corresponding to each upsampling process based on the target category information and target detection frame information corresponding to each upsampling process; Based on the position information, determining whether positions of multiple target objects corresponding to the multiple upsampling processes overlap; In response to the overlap of the positions, confidences corresponding to the multiple target objects with overlapping positions are determined, and the target object with the highest confidence is used as the final target object; The determined final target objects and target objects without position overlap are used as target objects in the video image.

9. A video processing device, characterized in that: include: An acquisition module is used to acquire the video to be processed; an identification module, used to identify at least one type of target object in each frame of the video to be processed by using a pre-trained target neural network, each type of target object being related to personal information; the target neural network includes a shared network and a plurality of branch networks, each of the branch networks being used to identify a type of target object; The recognition module, when using the pre-trained target neural network to recognize at least one type of target object in each frame of the video to be processed, is used to: Through the shared network, the video to be processed is subjected to video decoding processing to obtain each frame of video image corresponding to the video to be processed; and for each frame of the video image, the video image is subjected to multiple consecutive downsampling processing to obtain image feature information corresponding to each downsampling processing; wherein the input information of the subsequent downsampling processing in the multiple consecutive downsampling processing is the image feature information obtained by the previous downsampling processing, wherein the input information of the first downsampling processing is the video image; Performing multiple upsampling processes on the image feature information obtained by the last downsampling process, and obtaining initial category information corresponding to each upsampling process, and initial detection frame information corresponding to the initial category information; Using a target branch network among the multiple branch networks that matches the initial category information, perform multiple feature extractions on the initial category information corresponding to each upsampling process to obtain target category information, and perform multiple feature extractions on the initial detection frame information corresponding to the upsampling process to obtain target detection frame information; Determine at least one type of target object included in the video image based on the target category information and target detection frame information corresponding to each upsampling process; A processing module, used for performing blur processing on the target object in the identified video image to obtain a target image; The generating module is used to generate a blurred target video based on the target image corresponding to each frame of the video image.

10. A computer device, characterized in that: include: A processor and a memory, wherein the memory stores machine-readable instructions executable by the processor, and the processor is used to execute the machine-readable instructions stored in the memory. When the machine-readable instructions are executed by the processor, the processor executes the steps of the video processing method according to any one of claims 1 to 8.

11. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program. When the computer program is executed by a computer device, the computer device executes the steps of the video processing method according to any one of claims 1 to 8.

Images