An image processing method, apparatus, electronic device, and storage medium

Abstract

The application provides an image processing method and device, electronic equipment and storage medium, the method comprises: identifying a target object in an original image to obtain a plurality of identification boxes; segmenting the target object identified in each identification box to obtain a corresponding segmentation region; if the ratio of the size of the segmentation region corresponding to any identification box to the size of the identification box is less than a first preset proportion threshold, the region corresponding to the identification box is taken as a filling region; the union of all filling regions is taken as a target filling region of the image; when the ratio of the size of the target filling region to the size of the image is less than or equal to a second preset proportion threshold, the filling region of the image is filled. The application can ensure that the noise in the image is removed as much as possible.

Description

Technical Field

[0001] This invention relates to the field of image processing, and in particular to an image processing method, apparatus, electronic device, and storage medium. Background Technology

[0002] In some application scenarios, it is difficult to avoid pedestrians entering the frame when shooting real-world scenes. As a result, it is difficult to effectively model moving objects when using 3D methods for reconstruction. Therefore, it is necessary to remove pedestrians from the image before using 3D methods for reconstruction in order to achieve effective modeling of moving objects. Summary of the Invention

[0003] To address the aforementioned technical problems, the technical solution adopted by this invention is as follows:

[0004] According to a first aspect of the present invention, an image processing method is provided, the method comprising the following steps:

[0005] S100 identifies the target object in the original image and obtains m recognition boxes.

[0006] S200, the target object identified in the i-th recognition box is segmented to obtain the corresponding segmentation region, where i takes the value from 1 to m.

[0007] S300, if (SP) i / SF i If k > k1, the segmented region corresponding to the i-th recognition box is used as the corresponding filling region AF. i Otherwise, the region corresponding to the i-th recognition box is used as the corresponding fill region AF. i ; Obtain the filling regions AF1 to AF1 corresponding to m filling regions. m Among them, SP i Let SF be the size of the segmented region in the i-th recognition box. i is the size of the i-th recognition box; k1 is the first preset ratio threshold.

[0008] S400, obtain the overall fill region AF = (AF1∪AF2∪……∪AF) i ……∪AF m () is used as the target filling area of ​​the original image.

[0009] S500, if (SAF / S)≤k2, fill the AF in the original image to obtain the filled original image as the target image, where SAF is the size corresponding to AF, S is the size of the original image, and k2 is the second preset ratio threshold.

[0010] According to a second aspect of the present invention, an image processing apparatus is provided, the apparatus comprising:

[0011] The recognition module is used to identify target objects in the original image and obtain m recognition boxes, where i takes values ​​from 1 to m.

[0012] The segmentation module is used to segment the target object identified in the i-th recognition box to obtain the corresponding segmentation region.

[0013] The first acquisition module is used to obtain (SP) i / SF i When k > k1, the segmented region corresponding to the i-th recognition box is used as the corresponding filling region AF. i Otherwise, the region corresponding to the i-th recognition box is used as the corresponding fill region AF. i ; Obtain the filling regions AF1 to AF1 corresponding to m filling regions. m Among them, SP i Let SF be the size of the segmented region in the i-th recognition box. i is the size of the i-th recognition box; k1 is the first preset ratio threshold.

[0014] The second acquisition module is used to acquire the overall filled region AF = (AF1∪AF2∪……∪AF) i ……∪AF m () is used as the target filling area of ​​the original image.

[0015] A processing module is configured to fill the AF in the original image when (SAF / S)≤k2, obtaining the filled original image as the target image, where SAF is the size corresponding to AF, S is the size of the original image, and k2 is a second preset ratio threshold. According to a second aspect of the present invention, an electronic device is provided, including a processor and a memory; the processor executes the steps of the method described in the first aspect of the present invention by calling a program or instruction stored in the memory.

[0016] According to a second aspect of the present invention, a non-transitory computer-readable storage medium is provided, the non-transitory computer-readable storage medium storing a program or instructions that cause a computer to perform the steps of the method described in the first aspect of the present invention.

[0017] The present invention has at least the following beneficial effects:

[0018] The technical solution provided by this invention includes: identifying target objects in an original image to obtain multiple recognition boxes; segmenting the target objects identified in each recognition box to obtain corresponding segmented regions; if the ratio of the size of the segmented region corresponding to any recognition box to the size of the recognition box is less than a first preset ratio threshold, the region corresponding to the recognition box will be used as a filling region; the union of all filling regions will be used as the target filling region of the image; when the ratio of the size of the target filling region to the size of the image is less than or equal to a second preset ratio threshold, the filling region of the image will be filled. In this way, it can ensure that noise in the image is removed as much as possible, and images that are more suitable for model training can be selected.

[0019] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of the present invention, nor is it intended to limit the scope of the invention. Other features of the invention will become readily apparent from the following description. Attached Figure Description

[0020] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0021] Figure 1 A flowchart of an image processing method provided in an embodiment of the present invention. Detailed Implementation

[0022] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0023] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains. The terminology used herein in the description of this invention is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention. The term "and / or" as used herein includes any and all combinations of one or more of the associated listed items.

[0024] It should be noted that some exemplary embodiments are described as processes or methods depicted as flowcharts. Although the flowcharts describe the steps as sequential processes, many of these steps can be performed in parallel, concurrently, or simultaneously. Furthermore, the order of the steps can be rearranged. A process can be terminated when its operation is complete, but it may also have additional steps not included in the figures. A process can correspond to a method, function, procedure, subroutine, subroutine, etc.

[0025] This invention provides an image processing method for removing noise from images. In one illustrative embodiment, an application scenario can be a 3D reconstruction scenario, specifically removing target objects from images required for training a 3D GS (Gaussian Splatting) reconstruction model needed for 3D reconstruction of a target spatial environment. In this invention embodiment, the target object is a moving target not belonging to the target spatial environment, such as a moving human figure. In another illustrative embodiment, the target spatial environment can be an indoor environment.

[0026] like Figure 1 As shown, the method provided in this embodiment of the invention may include the following steps:

[0027] S100 identifies the target object in the original image and obtains m recognition boxes.

[0028] In this embodiment of the invention, the original image is an image obtained by taking a picture of the target spatial environment that needs to be reconstructed in 3D.

[0029] In this embodiment of the invention, existing object detection algorithms, such as the YOLO series object detection methods, can be used for identification. The identification box can be a bounding box that encloses the target object. Those skilled in the art will understand that any method that uses an object detection algorithm to identify a target object in an image and obtain the corresponding identification box falls within the protection scope of this invention.

[0030] In this embodiment of the invention, the target object can be an object specified by the user. In one illustrative embodiment, the target object can be a person and items carried by the person, such as a backpack, suitcase, etc.

[0031] S200, the target object identified in the i-th recognition box is segmented to obtain the corresponding segmentation region.

[0032] In this embodiment of the invention, existing image segmentation models can be used to segment the target object within the recognition box; for example, the SAM segmentation model can be used. Those skilled in the art will understand that any method using an image segmentation model to segment a target object falls within the scope of this invention.

[0033] S300, if (SP) i / SF i If k > k1, the segmented region corresponding to the i-th recognition box is used as the corresponding filling region AF. i Otherwise, the region corresponding to the i-th recognition box is used as the corresponding fill region AF. i ; Obtain the filling regions AF1 to AF1 corresponding to m filling regions. m Among them, SP i Let SF be the size of the segmented region in the i-th recognition box. i is the size of the i-th recognition box; i ranges from 1 to m; k1 is the first preset ratio threshold.

[0034] In this embodiment of the invention, the size of the segmented region and the size of the region corresponding to the recognition box can be the number of pixels contained in the corresponding region. For example, if the region contains Z pixels, then the size of the region is Z.

[0035] In this embodiment of the invention, if (SP) i / SF i If (SP) > k1, it indicates that the segmentation of the target object is good, and the target object can be completely or mostly identified. Therefore, the corresponding segmented region can be used as the filling region. If (SP) > k1, it means that the segmentation of the target object is good, and the target object can be completely or mostly identified. i / SF i If k ≤ k1, it indicates that the segmentation of the target object is poor, and only a small part of the target object is identified. The area corresponding to the recognition box can be used as the filling area to reduce the impact of large noise caused by poor segmentation quality on the subsequent 3D reconstruction model training. This is because if the segmentation is incomplete, the target object can still be observed during the subsequent 3D reconstruction model training process, which will affect the model training accuracy.

[0036] In one embodiment of the present invention, k1 can be obtained through the following steps:

[0037] S101, Obtain the sample image set IMG = {IMG1, IMG2, ..., IMG} j ..., IMG Q}, IMG i Let j be the j-th sample image in IMG, where j ranges from 1 to Q, and Q is the number of sample images.

[0038] In this embodiment of the invention, the sample image may be an image containing the target object.

[0039] S102, set variable r = 1.

[0040] S103, if r≤x, execute S104, otherwise execute S110; x is the number of preset segmentation models.

[0041] In this embodiment of the invention, the preset segmentation model can be an existing image segmentation model, and the specific value of x can be set based on actual needs, where x≥2.

[0042] S104, set j=1.

[0043] S105, if j≤Q, execute S106, then execute S109.

[0044] S106, for IMG j The target object in the image is identified, and multiple corresponding bounding boxes are obtained.

[0045] S107, using the r-th preset segmentation model to segment IMG j The target object identified in each corresponding recognition box is segmented to obtain the corresponding segmented region; thus, the IMG is obtained. j The corresponding segmented region set A r j ={A r j1 A r j2 , ..., A r jt , ..., A r jf(r，j)}, A r jt For A r j The t-th segment in the IMG, where t ranges from 1 to f(r,j), and f(r,j) is the IMG segment. j The number of corresponding segmented regions;

[0046] S108, Get IMG j The corresponding scale set K r j And obtain min(K) r j As IMG j The corresponding minimum segmentation ratio; set j = j + 1, execute S105; where K r j ={K r j1 K r j2 ..., K r jt ..., K r jf(r，j)}, K r jt For K r j The t-th proportion in K r jt = (SA)r j / SF r j SA r j For A r jt The corresponding size, SF r j For A r jt The size of the corresponding recognition box; min() means taking the minimum value.

[0047] S109, based on min(K) r 1) to min(K) r Q Obtain the target segmentation ratio KM corresponding to the r-th preset segmentation model. r Set r = r + 1 and execute S103.

[0048] In one illustrative embodiment of the present invention, KM r =(min(K) r 1)+min(K r 2) + ... + min(K) r j )+……+min(K r Q )) / Q, that is, take the average of all the minimum segmentation ratios corresponding to the segmentation model as the target segmentation ratio of the segmentation model.

[0049] In another illustrative embodiment of the present invention, KM r =min(min(K) r 1), min(K) r 2), ..., min(K) r j ), ..., min(K r Q That is, the smallest of all the minimum segmentation ratios corresponding to the segmentation model is taken as the target segmentation ratio of the segmentation model.

[0050] In another illustrative embodiment of the present invention, KM r =max(min(K) r 1), min(K) r 2), ..., min(K) r j ), ..., min(K r Q That is, the largest of all the minimum segmentation ratios corresponding to the segmentation model is taken as the target segmentation ratio of the segmentation model.

[0051] In another illustrative embodiment of the present invention, KM r It can be obtained through the following steps:

[0052] S1, obtain the average value AK r and deviation value DK r AK r =(min(K) r 1)+min(K r 2) + ... + min(K) r j )+……+min(K r Q )) / Q,

[0053] .

[0054] S2, iterate through the dataset (min(K) r 1), min(K) r 2), ..., min(K) r j ), ..., min(K r Q If |min(K) r j )-AK r |≤f*DK r , min(K r j ) is added as candidate data to the current candidate dataset, and the initial value of the current candidate dataset is empty; f is a preset coefficient, and in an illustrative embodiment, f = 2 or 3.

[0055] In this embodiment of the invention, if |min(K r j )-AK r |>f*DK r This indicates that min(K) r j ) indicates abnormal data; otherwise, it indicates normal data.

[0056] S3, obtain the average value of all candidate data in the current candidate dataset as KM. r .

[0057] In this embodiment, since the average of the proportions of abnormal data removed is used as the target segmentation proportion, the accuracy of the target segmentation proportion can be improved.

[0058] S110, based on KM1 to KM x Get k1.

[0059] In an illustrative embodiment of the present invention, k1 = (KM1 + KM2 + ... + KM) r +……+KM x k1 is the average of all minimum segmentation ratios corresponding to all segmentation models.

[0060] In another embodiment of the present invention, k1 = min(KM1, KM2, ..., KM) r ..., KM x ).

[0061] In another embodiment of the present invention, k1 = max(KM1, KM2, ..., KM) r ..., KM x ).

[0062] S400, obtain the overall fill region AF = (AF1∪AF2∪……∪AF) i ……∪AF m () is used as the target filling area of ​​the original image.

[0063] In this embodiment of the invention, since there may be some target objects that are not identified, for example, in some images there is a person carrying a backpack. The person and the backpack are identified by two boxes. Sometimes the person's box segmentation includes the backpack part, but the backpack box cannot identify the bag. Therefore, in order to ensure the integrity of the region, this invention first calculates the proportion of the segmented region in a single box to obtain the corresponding segmented region. Finally, the union of all segmented regions is taken as the target filling region, which can remove the target objects in the image as much as possible.

[0064] S500, if (SAF / S)≤k2, fill the AF in the original image to obtain the filled original image as the target image, where SAF is the size corresponding to AF, S is the size of the original image, and k2 is the second preset ratio threshold.

[0065] In this embodiment of the invention, the filling process is an operation of removing the target filling region from the image and repairing it. Existing methods can be used to fill the target filling region, for example, existing repair models such as the LaMa model can be used to repair the filling region.

[0066] In this embodiment of the invention, k2 may be an empirical value. In one illustrative embodiment, k2 is equal to one-quarter.

[0067] In this embodiment of the invention, if (SAF / S) ≤ k2, it indicates that the original image contains fewer target objects, relatively less noise, and has better image quality, making it suitable as a training set for denoising. If (SAF / S) > k2, it indicates that the original image contains more target objects, more noise, and has poorer image quality, making denoising unnecessary and unsuitable as a training set.

[0068] In this embodiment of the invention, SAF is equal to the number of pixels corresponding to AF, and S is equal to the number of pixels contained in the original image.

[0069] In summary, the image processing method provided by this embodiment of the invention, after identifying the target object, if the ratio of the size of the segmented region corresponding to any identification box to the size of the identification box is less than a first preset ratio threshold, then the region corresponding to the identification box will be used as a filling region, and the union of all filling regions will be used as the target filling region of the image. Furthermore, when the ratio of the size of the target filling region to the size of the image is less than or equal to a second preset ratio threshold, the filling region of the image will be filled. In this way, it can ensure that noise in the image is removed as much as possible, and images that are more suitable for model training can be selected, so as to improve the training accuracy of the model during model training.

[0070] Based on the same inventive concept, embodiments of the present invention provide an image processing apparatus, the apparatus comprising:

[0071] The recognition module is used to identify target objects in the original image and obtain m recognition boxes.

[0072] The segmentation module is used to segment the target object identified in the i-th recognition box to obtain the corresponding segmentation region, where i ranges from 1 to m.

[0073] The first acquisition module is used to obtain (SP) i / SF i When k > k1, the segmented region corresponding to the i-th recognition box is used as the corresponding filling region AF. i Otherwise, the region corresponding to the i-th recognition box is used as the corresponding fill region AF. i ; Obtain the filling regions AF1 to AF1 corresponding to m filling regions. m Among them, SP i Let SF be the size of the segmented region in the i-th recognition box. i is the size of the i-th recognition box; k1 is the first preset ratio threshold.

[0074] The second acquisition module is used to acquire the overall filled region AF = (AF1∪AF2∪……∪AF) i ……∪AF m() is used as the target filling area of ​​the original image.

[0075] The processing module is used to fill the AF in the original image when (SAF / S)≤k2, and obtain the filled original image as the target image. SAF is the size of the AF, S is the size of the original image, and k2 is the second preset ratio threshold.

[0076] This device can be used to perform Figure 1 The method shown in the illustrated embodiment is relevant here; therefore, the functions that each functional module of the device can achieve can be referred to. Figure 1 The embodiments shown are described in detail below.

[0077] This invention also provides an electronic device, including: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being configured to perform the method described in this invention.

[0078] This invention also provides a non-transitory computer-readable storage medium storing computer-executable instructions for performing the methods described in this invention.

[0079] It should be understood that the various forms of processes shown above can be used to reorder, add, or delete steps. For example, the steps described in this invention can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this invention can be achieved, and this is not limited herein.

[0080] The specific embodiments described above do not constitute a limitation on the scope of protection of this invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this invention should be included within the scope of protection of this invention.

Claims

1. An image processing method, characterized in that, The method includes the following steps: S100, identify the target object in the original image to obtain m recognition boxes; the target object is a moving target that does not belong to the target space environment; S200, segment the target object identified in the i-th recognition box to obtain the corresponding segmentation region; the value of i ranges from 1 to m; S300, if (SP) i / SF i If k > k1, the segmented region corresponding to the i-th recognition box is used as the corresponding filling region AF. i Otherwise, the region corresponding to the i-th recognition box is used as the corresponding fill region AF. i ; Obtain the filling regions AF1 to AF1 corresponding to m filling regions. m ; Among them, SP i Let SF be the size of the segmented region in the i-th recognition box. i Let k1 be the size of the i-th recognition box; k1 is the first preset proportional threshold. S400, obtain the total fill region AF = (AF1∪AF2∪……∪AF) i ……∪AF m ) as the target filling region of the original image; S500, if (SAF / S)≤k2, fill the AF in the original image to obtain the filled original image as the target image, where SAF is the size of the AF, S is the size of the original image, and k2 is the second preset ratio threshold. k1 is obtained through the following steps: S101, Obtain the sample image set IMG={IMG1, IMG2, ..., IMG...} j ..., IMG Q }, IMG j Let j be the j-th sample image in IMG, where j ranges from 1 to Q, and Q is the number of sample images; S102, set variable r=1; S103, if r≤x, execute S104; otherwise, execute S110; x is the number of preset segmentation models; S104, set j=1; S105, if j≤Q, execute S106, execute S109; S106, for IMG j The target object in the image is identified, and multiple corresponding bounding boxes are obtained. S107, using the r-th preset segmentation model to segment IMG j The target object identified in each corresponding recognition box is segmented to obtain the corresponding segmented region; thus, the IMG is obtained. j The corresponding segmented region set A r j ={A r j1 A r j2 , ..., A r jt , ..., A r jf（r，j） }, A r jt For A r j The t-th segment in the IMG, where t ranges from 1 to f(r,j), and f(r,j) is the IMG segment. j The number of corresponding segmented regions; S108, Get IMG j The corresponding scale set K r j And obtain min(K) r j As IMG j The corresponding minimum segmentation ratio; set j=j+1, execute S105; where K r j ={K r j1 K r j2 ..., K r jt ..., K r jf（r，j） }, K r jt For K r j The t-th proportion in K r jt = (SA) r j / SF r j ), SA r j For A r jt The corresponding size, SF r j For A r jt The corresponding size of the recognition box; min() means taking the minimum value; S109, based on min(K) r 1) to min(K) r Q Obtain the target segmentation ratio KM corresponding to the r-th preset segmentation model. r Set r = r + 1, and execute S103; S110, based on KM1 to KM x Get k1.

2. The method according to claim 1, characterized in that, KM r =(min)(K r 1)+min(K r 2)+……+min(K r j )+……+min(K Q r )) / Q。 3. The method according to claim 1, characterized in that, KM r It can be obtained through the following steps: S1, obtain the average value AK r and deviation value DK r AK r =(min(K r 1) + min(K) r 2) + ... + min(K) r j ) + ... + min(K r Q )) / Q, ; S2, traverse the dataset (min(K)) r 1), min(K) r 2), ..., min(K) r j ), ..., min(K) r Q If |min(K) r j )-AK r |≤f×DK r , min(K r j ) is added as candidate data to the current candidate dataset. The initial value of the current candidate dataset is empty, and f is a preset coefficient; S3, obtain the average value of all candidate data in the current candidate dataset as KM. r .

4. The method according to claim 1, characterized in that, k1=(KM1+KM2+......+KM r +......+KM x ) / x。 5. The method according to claim 1, characterized in that, The target objects include people and the items they carry.

6. An image processing apparatus, characterized in that, The device includes: The recognition module is used to identify target objects in the original image and obtain m recognition boxes; the value of i ranges from 1 to m; the target object is a moving target that does not belong to the target space environment; The segmentation module is used to segment the target object identified in the i-th recognition box to obtain the corresponding segmentation region; The first acquisition module is used to obtain information from (SP) i / SF i When k > k1, the segmented region corresponding to the i-th recognition box is used as the corresponding filling region AF. i Otherwise, the region corresponding to the i-th recognition box is used as the corresponding fill region AF. i ; Obtain the filling regions AF1 to AF1 corresponding to m filling regions. m Among them, SP i Let SF be the size of the segmented region in the i-th recognition box. i is the size of the i-th recognition box; k1 is the first preset proportional threshold; The second acquisition module is used to obtain the overall filled region AF = (AF1∪AF2∪……∪AF) i ……∪AF m ) as the target filling region of the original image; The processing module is used to fill the AF in the original image when (SAF / S)≤k2, and obtain the filled original image as the target image. SAF is the size corresponding to AF, S is the size of the original image, and k2 is the second preset ratio threshold. k1 is obtained through the following steps: S101, Obtain the sample image set IMG={IMG1, IMG2, ..., IMG...} j ..., IMG Q }, IMG j Let j be the j-th sample image in IMG, where j ranges from 1 to Q, and Q is the number of sample images; S102, set variable r=1; S103, if r≤x, execute S104; otherwise, execute S110; x is the number of preset segmentation models; S104, set j=1; S105, if j≤Q, execute S106, execute S109; S106, for IMG j The target object in the image is identified, and multiple corresponding bounding boxes are obtained. S107, using the r-th preset segmentation model to segment IMG j The target object identified in each corresponding recognition box is segmented to obtain the corresponding segmented region; thus, the IMG is obtained. j The corresponding segmented region set A r j ={A r j1 A r j2 , ..., A r jt , ..., A r jf（r，j） }, A r jt For A r j The t-th segment in the IMG, where t ranges from 1 to f(r,j), and f(r,j) is the IMG segment. j The number of corresponding segmented regions; S108, Get IMG j The corresponding scale set K r j And obtain min(K) r j As IMG j The corresponding minimum segmentation ratio; set j=j+1, execute S105; where K r j ={K r j1 K r j2 ..., K r jt ..., K r jf（r，j） }, K r jt For K r j The t-th proportion in K r jt = (SA) r j / SF r j ), SA r j For A r jt The corresponding size, SF r j For A r jt The corresponding size of the recognition box; min() means taking the minimum value; S109, based on min(K) r 1) to min(K) r Q Obtain the target segmentation ratio KM corresponding to the r-th preset segmentation model. r Set r = r + 1, and execute S103; S110, based on KM1 to KM x Get k1.

7. An electronic device, characterized in that, Including processor and memory; The processor executes the steps of the method as described in any one of claims 1 to 5 by invoking programs or instructions stored in the memory.

8. A non-transitory computer-readable storage medium, characterized in that, The non-transitory computer-readable storage medium is used to store a program or instructions that cause a computer to perform the steps of the method as described in any one of claims 1 to 5.

Images