Image background filtering method, system and storage medium
By training the object detection, foreground background distinction and depth estimation models, combined with non-maximum suppression and mask image analysis, the object detection problem with the similarity of the foreground and background features is solved, and the accurate identification and precise filtering of the foreground targets are achieved.
Patent Information
- Application Number
- CN202411381841.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-09-30
- Publication Date
- 2024-12-10
- Estimated Expiration
- 2044-09-30
AI Technical Summary
In object detection application scenarios, the local characteristics of the foreground target and the background target are similar, resulting in the confusion of positive and negative samples when distinguishing the model, reducing the detection accuracy.
The model is trained using the training data set, and the object detection model, the segmentation model and the depth estimation model are obtained respectively. Through the analysis of non-maximum suppression, depth feature map and mask image, the accurate prospect targets are gradually screened out. .
Effectively filter out the background, improve the detection accuracy of the foreground targets, and ensure that the foreground targets can be accurately identified under the situation of similar features in the background and foreground.
Smart Images
Figure CN118918330B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of image processing, and in particular to an image background filtering method, system and storage medium. Background Art
[0002] In target detection applications, it is often necessary to focus on the target in the field of view and eliminate the interference of the same target in the background. For visual solutions, other features are very similar to the features of non-background related targets in the field of view, and it is impossible to make a more effective distinction. If a target detection model is trained with a dataset that only annotates foreground targets, the local features of foreground targets and background targets are too similar, which will lead to confusion of positive and negative samples. The trained model will not only have no detection effect on distinguishing foreground targets from background targets, but also greatly reduce the detection accuracy.
[0003] In view of this, the present invention is proposed. Summary of the invention
[0004] The present invention aims to solve at least one of the above technical problems and provides an image background filtering method, system and storage medium.
[0005] To achieve the above object, the first technical solution provided by the present invention is:
[0006] Image background filtering method, including:
[0007] The model is trained using the training data set to obtain a target detection model, a foreground-background segmentation model, and a depth estimation model;
[0008] The target detection model is used to infer the image to be detected, and target frame information is obtained through non-maximum suppression. K target frames with the largest target areas are selected to obtain k targets, where k is a positive integer greater than or equal to 1.
[0009] Use the depth estimation model to infer the image to be detected to obtain a single-channel depth feature map, and determine a first target to be evaluated and a first foreground target from the k targets according to the depth values in the depth feature map;
[0010] The segmentation model for distinguishing foreground from background is used to infer the image to be detected to obtain a mask image, and a second target to be evaluated and a second foreground target are determined from the first target to be evaluated according to the foreground area ratio of the first target to be evaluated in the target coordinates corresponding to the mask image;
[0011] The corresponding foreground target evaluation values are calculated for the second target to be evaluated in turn, and a third foreground target is determined from the second target to be evaluated according to the foreground target evaluation values. The first foreground target, the second foreground target and the third foreground target are foreground targets determined after background filtering of the image to be detected.
[0012] Preferably, the training data set is a target detection data set based on a task scenario, a foreground and background segmentation data set based on a task scenario, and a depth estimation data set based on a task scenario.
[0013] Preferably, the method for determining the first target to be evaluated and the first foreground target from the k targets according to the depth value in the depth feature map is:
[0014] Traverse each pixel in the depth feature map and obtain the value of the pixel with the smallest depth as the zero point value;
[0015] Subtract the zero value from the depth value of each pixel in the depth feature map to obtain the relative depth value of each pixel;
[0016] Calculate the average depth value of each target among the k targets based on the relative depth values;
[0017] An object having an average depth value higher than a first average depth threshold and lower than a second average depth threshold is determined as a first object to be evaluated, and an object having an average depth value higher than the second average depth threshold is determined as a first foreground object, and the first average depth threshold is less than the second average depth threshold.
[0018] Preferably, the first average depth threshold is the average depth of the foreground objects in the training data set, and the second average depth threshold is the smaller value of the average depth of the foreground objects in the training data set minus the variance and the minimum depth of the foreground objects in the training data set.
[0019] Preferably, the mask image is smoothed.
[0020] Preferably, the method for determining the second target to be evaluated and the second foreground target from the first target to be evaluated according to the foreground area ratio of the first target to be evaluated in the target coordinates corresponding to the mask image is:
[0021] Calculate the foreground area ratio of each first target to be evaluated in the target coordinates corresponding to the mask image, determine the target whose foreground area ratio is higher than the first foreground area ratio threshold and lower than the second foreground area ratio threshold as the second target to be evaluated, and determine the target higher than the second foreground area ratio threshold as the second foreground target, and the first foreground area ratio threshold is less than the second foreground area ratio threshold.
[0022] Preferably, the calculation formula of the foreground target evaluation value is:
[0023] ;
[0024] Where P is the foreground target evaluation value, m and n are the width and height of the target respectively, i=0,1,…,m-1, j=0,1,…,n-1, B is the mask map output by the segmentation model for distinguishing the foreground and background, A is the depth feature map output by the depth estimation model, and x and y are the horizontal and vertical coordinates of the upper left corner of the target.
[0025] Preferably, the second target to be evaluated that is higher than a preset foreground target evaluation value is determined as the third foreground target.
[0026] The second technical solution provided by the present invention is:
[0027] Image background filtering system, including:
[0028] A training module is used to train the model using the training data set to obtain a target detection model, a foreground-background segmentation model, and a depth estimation model;
[0029] The first inference module is used to use the target detection model to infer the image to be detected, obtain target frame information through non-maximum suppression, select k target frames with the largest target area, and obtain k targets, where k is a positive integer greater than or equal to 1;
[0030] A second reasoning module uses the depth estimation model to reason about the image to be detected to obtain a single-channel depth feature map, and determines a first target to be evaluated and a first foreground target from the k targets according to the depth values in the depth feature map;
[0031] A third reasoning module is used to use the segmentation model for distinguishing foreground and background to infer the image to be detected to obtain a mask image, and determine a second target to be evaluated and a second foreground target from the first target to be evaluated according to the foreground area ratio of the first target to be evaluated in the target coordinates corresponding to the mask image;
[0032] A calculation module is used to calculate the corresponding foreground target evaluation value for the second target to be evaluated in turn, and determine the third foreground target from the second target to be evaluated according to the foreground target evaluation value, wherein the first foreground target, the second foreground target and the third foreground target are foreground targets determined after background filtering of the image to be detected.
[0033] The third technical solution provided by the present invention is:
[0034] A computer-readable storage medium stores a computer program, which is executed by a processor to implement any image background filtering method described in the first technical solution.
[0035] Compared with the prior art, the present invention has the following beneficial effects:
[0036] The present invention trains models for target, foreground-background segmentation and image depth, and uses these three models to infer the detection model in turn, and continuously filters and screens the inference results of the previous layer to finally obtain an accurate foreground target. When there are multiple similar features in the background and foreground images, the present invention accurately filters out the background and obtains an accurate foreground target. BRIEF DESCRIPTION OF THE DRAWINGS
[0037] Figure 1 A schematic diagram of a process flow of an image background filtering method provided by an embodiment of the present invention;
[0038] Figure 2 A schematic diagram of the structure of an image background filtering system 200 provided in an embodiment of the present invention. DETAILED DESCRIPTION
[0039] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0040] The first embodiment of the present invention provides an image background filtering method. Figure 1 A schematic flow chart of an image background filtering method provided in an embodiment of the present invention.
[0041] like Figure 1 As shown, the image background filtering method specifically includes the following steps:
[0042] S101, training the model using the training data set to obtain a target detection model, a foreground-background segmentation model, and a depth estimation model.
[0043] It can be understood that the training data set is a target detection data set based on a task scenario, a foreground and background segmentation data set based on a task scenario, and a depth estimation data set based on a task scenario.
[0044] The models used in the embodiments of the present invention are the target detection models, foreground-background segmentation models, and depth estimation models commonly used in the field. For example, the target detection model can use YOLO, SSD, R-CNN, etc.; the foreground-background segmentation model can use DeepLab, HRnet, Unet, etc.; the depth estimation model can use DeepLabDepth, MobileNetMonodepth, MonoDepth, etc.
[0045] S102, use the target detection model to infer the image to be detected, obtain target frame information through non-maximum suppression, select k target frames with the largest target area, and obtain k targets, where k is a positive integer greater than or equal to 1.
[0046] It should be noted that the image to be detected includes a video frame, which is a special form of an image.
[0047] It should also be noted that in the target detection of this step in the embodiment of the present invention, the resolution of the depth feature map in S103 and the mask image in S104 are the same, the coordinates are shared, and both are calculated through the target detection frame.
[0048] This step is equivalent to the first screening, which preliminarily screens out the targets. Non-maximum suppression is a conventional post-processing of target detection. The target frame information obtained is multiple targets obtained by the target detection task. k is a positive integer greater than or equal to 1, which can be determined by technicians in this field based on the number of task scenarios. For example, in a live broadcast of a sports event, the foreground is generally only the order of magnitude of two or three athletes and coaches, and there are a large number of spectators in the background. In this case, k can be set to 10 for the first screening.
[0049] S103, using the depth estimation model to infer the image to be detected to obtain a single-channel depth feature map, and determining a first target to be evaluated and a first foreground target from the k targets according to the depth values in the depth feature map.
[0050] It can be understood that the output of the depth estimation model is a single-channel grayscale image with the same size as the original image, whose value is the depth information, that is, the depth feature map is consistent with the size of the image to be detected.
[0051] The specific method of this step is:
[0052] Traverse each pixel in the depth feature map and obtain the value of the pixel with the smallest depth as the zero point value;
[0053] Subtract the zero value from the depth value of each pixel in the depth feature map to obtain the relative depth value of each pixel;
[0054] Calculate the average depth value of each target among k targets based on the relative depth value;
[0055] The target whose average depth value is higher than a first average depth threshold and lower than a second average depth threshold is determined as the first target to be evaluated, and the target whose average depth value is higher than the second average depth threshold is determined as the first foreground target, and the first average depth threshold is less than the second average depth threshold.
[0056] In addition, targets whose average depth values are lower than a first average depth threshold are determined as background targets.
[0057] Among them, the first average depth threshold and the second average depth threshold can be set by those skilled in the art according to the task scenario, and can be set based on the depth information of the foreground and background targets of the task training data set or directly based on the specific usage scenario. Generally speaking, the first average depth threshold is the average depth of the foreground target of the training data set; the second average depth threshold is the smaller value of the average depth of the foreground target of the training data set minus the variance and the minimum depth of the foreground target of the training data set, that is, the smaller value of the two is taken.
[0058] S104, using the foreground-background distinguishing segmentation model to infer the image to be detected to obtain a mask image, and determining a second target to be evaluated and a second foreground target from the first target to be evaluated according to the foreground area ratio of the first target to be evaluated in the target coordinates corresponding to the mask image.
[0059] This step is equivalent to secondary filtering. The segmentation model for distinguishing the foreground and background outputs a mask image that is consistent with the size of the original image. The mask image can distinguish between the background and the foreground. The mask image is such as 0101, where 1 is the background and 0 is the foreground. However, the accuracy of the mask image will not be completely accurate, and the target is a rectangular frame, part of which is non-target. The non-target may be the foreground or the background. Therefore, this step determines whether it belongs to the foreground or the background by the area ratio of the foreground and background of the first target to be evaluated.
[0060] Specifically, the steps are:
[0061] Calculate the foreground area ratio of each first target to be evaluated in the target coordinates corresponding to the mask image, determine the target whose foreground area ratio is higher than the first foreground area ratio threshold and lower than the second foreground area ratio threshold as the second target to be evaluated, and determine the target higher than the second foreground area ratio threshold as the second foreground target, and the first foreground area ratio threshold is less than the second foreground area ratio threshold.
[0062] In addition, targets with an area ratio lower than the first foreground area ratio are determined as background targets.
[0063] In some preferred implementations, the mask image is smoothed before determining the second target to be evaluated.
[0064] S105, calculating the corresponding foreground target evaluation values for the second target to be evaluated in turn, and determining the third foreground target from the second target to be evaluated according to the foreground target evaluation values, wherein the first foreground target, the second foreground target and the third foreground target are foreground targets determined after background filtering of the image to be detected.
[0065] This step is equivalent to three filterings. The second target to be evaluated that is higher than the preset foreground target evaluation value is determined as the third foreground target.
[0066] The foreground target obtained after image background filtering in the embodiment of the present invention is a set of the first foreground target, the second foreground target and the third foreground target.
[0067] In some preferred implementations, the calculation formula for the foreground target evaluation value is:
[0068] ;
[0069] Among them, P is the foreground target evaluation value, m and n are the width and height of the target respectively, i=0,1,…,m-1, j=0,1,…,n-1, B is the mask map output by the segmentation model for distinguishing the foreground and background, A is the depth feature map output by the depth estimation model, and x and y are the horizontal and vertical coordinates of the upper left point of the target. It should be noted that when the coordinate of the mask map is 0, it means that the coordinate is foreground, and when the coordinate of the mask map is 1, it means that the coordinate is background; each coordinate in the depth feature map is depth information.
[0070] A second aspect of the present invention provides an image background filtering system. Figure 2 A schematic diagram of the structure of an image background filtering system 200 provided in an embodiment of the present invention.
[0071] like Figure 2 As shown, the image background filtering system 200 provided by the present invention comprises a training module 201, a first reasoning module 202, a second reasoning module 203, a third reasoning module 204, and a calculation module 205. The detailed functions of each module are as follows.
[0072] A training module 201 is used to train the model using a training data set to obtain a target detection model, a foreground-background segmentation model, and a depth estimation model;
[0073] The first reasoning module 202 is used to use the target detection model to reason about the image to be detected, obtain target frame information through non-maximum suppression, select k target frames with the largest target area, and obtain k targets, where k is a positive integer greater than or equal to 1;
[0074] The second reasoning module 203 uses the depth estimation model to reason the image to be detected to obtain a single-channel depth feature map, and determines the first target to be evaluated and the first foreground target from the k targets according to the depth values in the depth feature map;
[0075] A third reasoning module 204 is used to use the segmentation model for distinguishing foreground and background to infer the image to be detected to obtain a mask image, and determine a second target to be evaluated and a second foreground target from the first target to be evaluated according to the foreground area ratio of the first target to be evaluated in the target coordinates corresponding to the mask image;
[0076] The calculation module 205 is used to calculate the corresponding foreground target evaluation value for the second target to be evaluated in turn, and determine the third foreground target from the second target to be evaluated according to the foreground target evaluation value. The first foreground target, the second foreground target and the third foreground target are foreground targets determined after the image to be detected is filtered through the background.
[0077] Based on the above embodiments, a third aspect of the present invention further provides a computer-readable storage medium storing a computer program, wherein the computer program is used to enable the computer to execute the image background filtering method disclosed in the embodiments of the present invention.
[0078] Based on the above embodiments, the present invention further provides a computer program product, including a computer program, which implements the image background filtering method disclosed in the embodiments of the present invention when executed by a processor.
[0079] The computer program includes computer program code, which may be in source code form, object code form, executable file or some intermediate form, etc. The computer readable medium may include: any entity or device capable of carrying the computer program code, recording medium, USB flash drive, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signal, telecommunication signal and software distribution medium, etc. It should be noted that the content contained in the computer readable medium may be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer readable media do not include electric carrier signals and telecommunication signals.
[0080] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations may include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.
[0081] In the several embodiments provided in the present application, it should be understood that the disclosed devices, apparatuses and methods can be implemented in other ways. For example, the device embodiments described above are only schematic, for example, the division of units is only a logical function division, and there may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.
[0082] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.
[0083] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.
[0084] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods of each embodiment of the present application.
[0085] The above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, a person skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application.
Claims
1. An image background filtering method, characterized in that: Include: The model is trained using the training data set to obtain a target detection model, a foreground-background segmentation model, and a depth estimation model. The target detection model is used to infer the image to be detected, and target frame information is obtained through non-maximum suppression. K target frames with the largest target areas are selected to obtain k targets, where k is a positive integer greater than or equal to 1. The depth estimation model is used to infer the image to be detected to obtain a single-channel depth feature map, and each pixel in the depth feature map is traversed to obtain the value of the pixel with the smallest depth as the zero point value; the depth value of each pixel in the depth feature map is respectively subtracted from the zero point value to obtain the relative depth value of each pixel; the average depth value of each target among the k targets is calculated according to the relative depth value; the target whose average depth value is higher than the first average depth threshold and lower than the second average depth threshold is determined as the first target to be evaluated, and the target whose average depth value is higher than the second average depth threshold is determined as the first foreground target, and the first average depth threshold is less than the second average depth threshold; The foreground-background-distinguishing segmentation model is used to infer the image to be detected to obtain a mask image, and the foreground area ratio of each first target to be evaluated in the target coordinates corresponding to the mask image is calculated, and the target whose foreground area ratio is higher than the first foreground area ratio threshold and lower than the second foreground area ratio threshold is determined as the second target to be evaluated, and the target whose foreground area ratio is higher than the second foreground area ratio threshold is determined as the second foreground target, and the first foreground area ratio threshold is less than the second foreground area ratio threshold; Calculating the corresponding foreground target evaluation values for the second target to be evaluated in sequence, and determining a third foreground target from the second target to be evaluated according to the foreground target evaluation values, wherein the first foreground target, the second foreground target and the third foreground target are foreground targets determined after background filtering of the image to be detected; The calculation formula for the prospect target evaluation value is: ; Where P is the foreground target evaluation value, m and n are the width and height of the target respectively, i=0,1,…,m-1, j=0,1,…,n-1, B is the mask map output by the segmentation model for distinguishing the foreground and background, A is the depth feature map output by the depth estimation model, and x and y are the horizontal and vertical coordinates of the upper left corner of the target.
2. The image background filtering method according to claim 1, characterized in that: The training data set includes a target detection data set based on a task scenario, a foreground and background segmentation data set based on a task scenario, and a depth estimation data set based on a task scenario.
3. The image background filtering method according to claim 1, characterized in that: The first average depth threshold is the average depth of the foreground objects in the training data set, and the second average depth threshold is the smaller value of the average depth of the foreground objects in the training data set minus the variance and the minimum depth of the foreground objects in the training data set.
4. The image background filtering method according to claim 1, characterized in that: The mask image is smoothed.
5. The image background filtering method according to claim 1, characterized in that: The second target to be evaluated that is higher than the preset foreground target evaluation value is determined as the third foreground target.
6. Image background filtering system, characterized in that: Include: A training module is used to train the model using the training data set to obtain a target detection model, a foreground-background segmentation model, and a depth estimation model; The first inference module is used to use the target detection model to infer the image to be detected, obtain target frame information through non-maximum suppression, select k target frames with the largest target area, and obtain k targets, where k is a positive integer greater than or equal to 1; The second reasoning module uses the depth estimation model to reason about the image to be detected to obtain a single-channel depth feature map, traverses each pixel in the depth feature map, obtains the value of the pixel with the smallest depth as the zero point value; subtracts the zero point value from the depth value of each pixel in the depth feature map to obtain the relative depth value of each pixel; calculates the average depth value of each target among the k targets according to the relative depth value; determines the target with an average depth value higher than a first average depth threshold and lower than a second average depth threshold as the first target to be evaluated, and determines the target higher than the second average depth threshold as the first foreground target, and the first average depth threshold is less than the second average depth threshold; A third reasoning module is used to use the segmentation model for distinguishing the foreground and background to infer the image to be detected to obtain a mask image, calculate the foreground area ratio of each first target to be evaluated in the target coordinates corresponding to the mask image, determine the target with a foreground area ratio higher than a first foreground area ratio threshold and lower than a second foreground area ratio threshold as the second target to be evaluated, and determine the target higher than the second foreground area ratio threshold as the second foreground target, and the first foreground area ratio threshold is lower than the second foreground area ratio threshold; a calculation module, configured to calculate the corresponding foreground target evaluation values for the second target to be evaluated in sequence, and determine the third foreground target from the second target to be evaluated according to the foreground target evaluation values, wherein the first foreground target, the second foreground target and the third foreground target are foreground targets determined after the image to be detected is filtered through the background; The calculation formula for the prospect target evaluation value is: ; Where P is the foreground target evaluation value, m and n are the width and height of the target respectively, i=0,1,…,m-1, j=0,1,…,n-1, B is the mask map output by the segmentation model for distinguishing the foreground and background, A is the depth feature map output by the depth estimation model, and x and y are the horizontal and vertical coordinates of the upper left corner of the target.
7. A computer-readable storage medium having a computer program stored thereon, characterized in that: The program is executed by a processor to implement the image background filtering method as described in any one of claims 1 to 5.