Image processing method and device, electronic equipment and storage medium
By identifying and segmenting the image, judging and filling the image, the noise problem caused by pedestrians entering the lens is solved, and the training accuracy of the 3D reconstruction model is improved.
Patent Information
- Application Number
- CN202410900330.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-05
- Publication Date
- 2024-11-01
- Estimated Expiration
- 2044-07-05
AI Technical Summary
When reconstructing using 3D methods, pedestrians are inevitably allowed to enter the camera, which leads to difficulty in effectively modeling mobile items. Pedestrians in the picture need to be removed to achieve effective modeling.
By identifying and segmenting the original image, whether to fill is performed based on the size of the segmented area and the size of the recognition box, the overall fill area is obtained and fill is performed to remove noise, ensuring that the image quality is suitable for model training.
Effectively remove noise in the image and select images that are more suitable for model training, thereby improving the training accuracy of 3D reconstruction models.
Smart Images

Figure CN118887232A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of image processing, and in particular to an image processing method, device, electronic equipment and storage medium. Background Art
[0002] In some application scenarios, when shooting real scenes, it is difficult to avoid pedestrians entering the lens. Therefore, when using 3D methods for reconstruction, it is difficult to effectively model moving objects. Therefore, it is necessary to remove pedestrians from the picture and then use 3D methods for reconstruction to achieve effective modeling of moving objects. Summary of the invention
[0003] In view of the above technical problems, the technical solution adopted by the present invention is:
[0004] According to a first aspect of the present invention, there is provided an image processing method, the method comprising the following steps:
[0005] S100, identifying the target object in the original image to obtain m identification frames.
[0006] S200, segmenting the target object identified in the i-th identification frame to obtain a corresponding segmented area, where the value of i ranges from 1 to m.
[0007] S300, if (SP i / SF i )>k1, the corresponding segmented area in the i-th recognition frame is used as the corresponding filling area AF i Otherwise, the area corresponding to the i-th recognition box is used as the corresponding filling area AF i ; Get the filling areas AF1 to AF corresponding to the m filling areas m Among them, SP i is the size of the segmented area in the i-th recognition frame, SF i is the size of the i-th recognition box; k1 is the first preset ratio threshold.
[0008] S400, obtaining the overall filling area AF = (AF1∪AF2∪…∪AF i ……∪AF m ) as the target filling area of the original image.
[0009] S500, if (SAF / S)≤k2, fill the AF in the original image to obtain the original image after filling as the target image, SAF is the size corresponding to AF, S is the size of the original image, and k2 is the second preset ratio threshold.
[0010] According to a second aspect of the present invention, there is provided an image processing apparatus, the apparatus comprising:
[0011] The recognition module is used to recognize the target object in the original image and obtain m recognition boxes, where the value of i ranges from 1 to m.
[0012] The segmentation module is used to segment the target object identified in the i-th recognition frame to obtain the corresponding segmentation area.
[0013] The first acquisition module is used to obtain i / SF i )>k1, the corresponding segmented area in the i-th recognition frame is used as the corresponding filling area AF i Otherwise, the area corresponding to the i-th recognition box is used as the corresponding filling area AF i ; Get the filling areas AF1 to AF corresponding to the m filling areas m Among them, SP i is the size of the segmented area in the i-th recognition frame, SF i is the size of the i-th recognition box; k1 is the first preset ratio threshold.
[0014] The second acquisition module is used to obtain the overall filling area AF = (AF1∪AF2∪…∪AF i ……∪AF m ) as the target filling area of the original image.
[0015] The processing module is used to fill the AF in the original image when it is determined that (SAF / S)≤k2, and obtain the original image after filling as the target image, SAF is the size corresponding to AF, S is the size of the original image, and k2 is a second preset ratio threshold. According to a second aspect of the present invention, an electronic device is provided, including a processor and a memory; the processor is used to execute the steps of the method described in the first aspect of the present invention by calling a program or instruction stored in the memory.
[0016] According to a second aspect of the present invention, there is provided a non-transitory computer-readable storage medium storing a program or instructions, wherein the program or instructions enable a computer to execute the steps of the method described in the first aspect of the present invention.
[0017] The present invention has at least the following beneficial effects:
[0018] The technical solution provided by the embodiment of the present invention includes: identifying the target object in the original image to obtain multiple identification frames; segmenting the target object identified in each identification frame to obtain the corresponding segmented area; if the ratio of the size of the segmented area corresponding to any identification frame to the size of the identification frame is less than a first preset ratio threshold, the area corresponding to the identification frame will be used as a filling area; the union of all the filling areas will be used as the target filling area of the image; when the ratio of the size of the target filling area to the size of the image is less than or equal to a second preset ratio threshold, the filling area of the image is filled. In this way, it can be ensured that the noise in the image is removed as much as possible, and an image that is more suitable for model training can be selected.
[0019] It should be understood that the contents described in this section are not intended to identify the key or important features of the embodiments of the present invention, nor are they intended to limit the scope of the present invention. Other features of the present invention will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS
[0020] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.
[0021] Figure 1 The present invention provides a flowchart of an image processing method. DETAILED DESCRIPTION
[0022] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.
[0023] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art of the present invention. The terms used herein in the specification of the present invention are only for the purpose of describing specific embodiments and are not intended to limit the present invention. The term "and / or" used herein includes any and all combinations of one or more related listed items.
[0024] It should be noted that some exemplary embodiments are described as processes or methods depicted as flow charts. Although the flow charts describe the steps as sequential processes, many of the steps can be implemented in parallel, concurrently or simultaneously. In addition, the order of the steps can be rearranged. The process can be terminated when its operation is completed, but it can also have additional steps not included in the accompanying drawings. The process can correspond to a method, function, procedure, subroutine, subprogram, etc.
[0025] An embodiment of the present invention provides an image processing method for removing noise from an image. In an illustrative embodiment, an application scenario may be a 3D reconstruction scenario, specifically, removing a target object in an image required for training a 3D GS (Gaussian Splatting) reconstruction model required for 3D reconstruction of a target space environment. In an embodiment of the present invention, the target object is a moving target that does not belong to the target space environment, such as a moving human figure. In an illustrative embodiment of the present invention, the target space environment may be an indoor environment.
[0026] like Figure 1 As shown, the method provided by the embodiment of the present invention may include the following steps:
[0027] S100, identifying the target object in the original image to obtain m identification frames.
[0028] In the embodiment of the present invention, the original image is an image obtained by photographing the target space environment that needs to be 3D reconstructed.
[0029] In an embodiment of the present invention, an existing target detection algorithm such as the YOLO series target detection method can be used for identification. The identification frame can be a bounding box that surrounds the target object. It is known to those skilled in the art that any method of identifying the target object in the image using a target detection algorithm to obtain a corresponding identification frame belongs to the protection scope of the present invention.
[0030] In an embodiment of the present invention, the target object may be an object specified by a user. In an illustrative embodiment, the target object may be a person and articles carried by the person, such as a backpack, a suitcase, etc.
[0031] S200, segmenting the target object identified in the i-th identification frame to obtain a corresponding segmented area.
[0032] In the embodiment of the present invention, the target object in the recognition frame can be segmented using an existing image segmentation model, for example, the target object can be segmented using a SAM segmentation model. It is known to those skilled in the art that any method for segmenting the target object using an image segmentation model falls within the protection scope of the present invention.
[0033] S300, if (SP i / SF i )>k1, the corresponding segmented area in the i-th recognition frame is used as the corresponding filling area AF i Otherwise, the area corresponding to the i-th recognition box is used as the corresponding filling area AF i ; Get the filling areas AF1 to AF corresponding to the m filling areas m Among them, SP i is the size of the segmented area in the i-th recognition frame, SF i is the size of the i-th recognition box; the value of i ranges from 1 to m; k1 is the first preset ratio threshold.
[0034] In the embodiment of the present invention, the size of the segmented area and the size of the area corresponding to the identification box may be the number of pixels contained in the corresponding area. For example, if the area contains Z pixels, the size of the area is Z.
[0035] In the embodiment of the present invention, if (SP i / SF i )>k1, it means that the segmentation of the target object is good and the target object can be completely or mostly recognized. Therefore, the corresponding segmented area can be used as the filling area. i / SF i )≤k1, it means that the segmentation of the target object is poor, and only a small part of the target object is recognized. The area corresponding to the corresponding recognition box can be used as a filling area to reduce the influence of the large noise caused by the poor segmentation quality on the subsequent 3D reconstruction model training. Because, if the segmentation is incomplete, the target object can still be observed in the subsequent 3D reconstruction model training process, which will affect the model training accuracy.
[0036] In one embodiment of the present invention, k1 can be obtained by the following steps:
[0037] S101, obtain a sample image set IMG = {IMG1, IMG2, ..., IMG j , ..., IMG Q},IMG i is the j-th sample image in IMG, the value of j ranges from 1 to Q, and Q is the number of sample images.
[0038] In the embodiment of the present invention, the sample image may be an image containing a target object.
[0039] S102, set variable r=1.
[0040] S103, if r≤x, execute S104, otherwise, execute S110; x is the number of preset segmentation models.
[0041] In the embodiment of the present invention, the preset segmentation model may be an existing image segmentation model, and the specific value of x may be set based on actual needs, where x≥2.
[0042] S104, set j=1.
[0043] S105, if j≤Q, execute S106, execute S109.
[0044] S106, to IMG j The target object in the image is identified to obtain multiple corresponding recognition boxes.
[0045] S107, using the rth preset segmentation model to segment IMG j The target object identified in each corresponding recognition frame is segmented to obtain the corresponding segmented area; IMG j The corresponding segmentation area set A r j ={A r j1 , A r j2 , ..., A r jt , ..., A r jf(r,j)}, A r jt A r j The t-th segmentation region in , t ranges from 1 to f(r, j), f(r, j) is IMG j The number of corresponding segmented regions;
[0046] S108, Get IMG j The corresponding ratio set K r j And get min(K r j ) as IMG j The corresponding minimum segmentation ratio; set j = j + 1, and execute S105; wherein, K r j ={K r j1 , K r j2 , ..., K r jt , ..., K r jf(r,j)}, K r jt K r j The tth proportion in K r jt =(SAr j / SF r j ), SA r j A r jt Corresponding size, SF r j A r jt The size of the corresponding recognition box; min() means taking the minimum value.
[0047] S109, based on min(K r 1) to min(K r Q ) Get the target segmentation ratio KM corresponding to the rth preset segmentation model r ; Set r=r+1 and execute S103.
[0048] In an exemplary embodiment of the present invention, KM r =(min(K r 1)+min(K r 2)+……+min(K r j )+……+min(K r Q )) / Q, that is, take the average of all minimum segmentation ratios corresponding to the segmentation model as the target segmentation ratio of the segmentation model.
[0049] In another exemplary embodiment of the present invention, KM r =min(min(K r 1), min(K r 2), ..., min(K r j ), ..., min(K r Q )), that is, take the smallest of all minimum segmentation ratios corresponding to the segmentation model as the target segmentation ratio of the segmentation model.
[0050] In another exemplary embodiment of the present invention, KM r =max(min(K r 1), min(K r 2), ..., min(K r j ), ..., min(K r Q )), that is, take the maximum of all minimum segmentation ratios corresponding to the segmentation model as the target segmentation ratio of the segmentation model.
[0051] In another exemplary embodiment of the present invention, KM r It can be obtained through the following steps:
[0052] S1, get the average value AK r and deviation value DK r , A.K. r =(min(K r 1)+min(K r 2)+……+min(K r j )+……+min
[0053]
[0054] S2, traverse the data set (min(K r 1), min(K r 2), ..., min(K r j ), ..., min(K r Q )), if |min(K r j )-AK r ∣≤f*DK r , min(K r j ) is added as candidate data to the current candidate data set, and the initial value of the current candidate data set is a null value; f is a preset coefficient, and in an illustrative embodiment, f=2 or 3.
[0055] In the embodiment of the present invention, if |min(K r j )-AK r ∣>f*DK r , indicating that min(K r j ) is abnormal data, otherwise, it is normal data.
[0056] S3, get the average value of all candidate data in the current candidate data set as KM r .
[0057] In this embodiment, since the mean value of the ratio of excluding abnormal data is used as the target segmentation ratio, the accuracy of the target segmentation ratio can be improved.
[0058] S110, based on KM1 to KM x Get k1.
[0059] In an exemplary embodiment of the present invention, k1=(KM1+KM2+...+KM r +……+KM x) / x, that is, take the mean of all minimum segmentation ratios corresponding to all segmentation models as k1.
[0060] In another embodiment of the present invention, k1=min(KM1, KM2, ..., KM r , ..., KM x ).
[0061] In another embodiment of the present invention, k1=max(KM1, KM2, ..., KM r , ..., KM x ).
[0062] S400, obtaining the overall filling area AF = (AF1∪AF2∪…∪AF i ……∪AF m ) as the target filling area of the original image.
[0063] In the embodiments of the present invention, since there may be some target objects that are not recognized, for example, in some images there are people carrying backpacks, the person and the backpack are identified as two frames, and sometimes the segmentation of the person frame includes the backpack part, but the backpack frame cannot recognize the bag. Therefore, in order to ensure the integrity of the area, the present invention first calculates the proportion of the segmented area in a single frame to obtain the corresponding segmented area, and finally takes the union of all the segmented areas as the target filling area, which can remove the target object in the image as much as possible.
[0064] S500, if (SAF / S)≤k2, fill the AF in the original image to obtain the original image after filling as the target image, SAF is the size corresponding to AF, S is the size of the original image, and k2 is the second preset ratio threshold.
[0065] In an embodiment of the present invention, the filling process is an operation of removing the target filling area from the image and patching it. The target filling area may be filled using existing methods, for example, the filling area may be patched using an existing patching model such as the LaMa model.
[0066] In the embodiment of the present invention, k2 may be an empirical value. In an exemplary embodiment, k2 is equal to one quarter.
[0067] In the embodiment of the present invention, if (SAF / S)≤k2, it means that there are fewer target objects in the original image, the noise is relatively small, the image quality is good, and it can be used as a training set and denoising can be performed on it. If (SAF / S)>k2, it means that there are more target objects in the original image, the noise is more, the image quality is poor, and there is no need to denoise it, and it is not suitable for a training set.
[0068] In the embodiment of the present invention, SAF is equal to the number of pixel points corresponding to AF, and S is equal to the number of pixel points included in the original image.
[0069] In summary, the image processing method provided by the embodiment of the present invention, after identifying the target object, if the ratio of the size of the segmented area corresponding to any identification box to the size of the identification box is less than a first preset ratio threshold, the area corresponding to the identification box will be used as the filling area, and the union of all the filling areas will be used as the target filling area of the image, and when the ratio of the size of the target filling area to the size of the image is less than or equal to a second preset ratio threshold, the filling area of the image is filled. In this way, it can be ensured that the noise in the image is removed as much as possible, and an image that is more suitable for model training can be selected, so as to improve the training accuracy of the model during model training.
[0070] Based on the same inventive concept, an embodiment of the present invention provides an image processing device, the device comprising:
[0071] The recognition module is used to recognize the target object in the original image and obtain m recognition boxes.
[0072] The segmentation module is used to segment the target object identified in the i-th recognition frame to obtain the corresponding segmentation area, and the value of i ranges from 1 to m.
[0073] The first acquisition module is used to obtain i / SF i )>k1, the corresponding segmented area in the i-th recognition frame is used as the corresponding filling area AF i Otherwise, the area corresponding to the i-th recognition box is used as the corresponding filling area AF i ; Get the filling areas AF1 to AF corresponding to the m filling areas m Among them, SP i is the size of the segmented area in the i-th recognition frame, SF i is the size of the i-th recognition box; k1 is the first preset ratio threshold.
[0074] The second acquisition module is used to obtain the overall filling area AF = (AF1∪AF2∪…∪AF i ……∪AF m ) as the target filling area of the original image.
[0075] The processing module is used to fill the AF in the original image when it is determined that (SAF / S)≤k2, and obtain the original image after the filling process as the target image, SAF is the size corresponding to AF, S is the size of the original image, and k2 is the second preset ratio threshold.
[0076] The device can be used to perform Figure 1 Therefore, for the functions that can be realized by each functional module of the device, please refer to Figure 1 The description of the illustrated embodiment is not repeated in detail.
[0077] An embodiment of the present invention also provides an electronic device, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are configured to execute the method described in the embodiment of the present invention.
[0078] The embodiment of the present invention further provides a non-transitory computer-readable storage medium storing computer-executable instructions, wherein the computer instructions are used to execute the method described in the embodiment of the present invention.
[0079] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps described in the present invention can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solution disclosed in the present invention can be achieved, and this document does not limit this.
[0080] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.
Claims
1. An image processing method, It is characterized in that The method comprises the following steps: S100, identifying the target object in the original image to obtain m identification frames; S200, segmenting the target object identified in the i-th identification frame to obtain a corresponding segmented area; the value of i ranges from 1 to m; S300, if (SP i / SF i )>k1, the corresponding segmented area in the i-th recognition frame is used as the corresponding filling area AF i Otherwise, the area corresponding to the i-th recognition box is used as the corresponding filling area AF i ; Get the filling area AF corresponding to the m filling areas 1 To AF m ; Among them, SP i is the size of the segmented area in the i-th recognition frame, SF i is the size of the i-th recognition box; k1 is the first preset ratio threshold; S400, obtaining the overall filling area AF = (AF 1 ∪AF 2 ∪……∪AF i ……∪AF m ) as the target filling area of the original image; S500, if (SAF / S)≤k2, fill the AF in the original image to obtain the original image after filling as the target image, SAF is the size corresponding to AF, S is the size of the original image, and k2 is the second preset ratio threshold.
2. The method according to claim 1, It is characterized in that k1 is obtained by the following steps: S101, obtain sample image set IMG = {IMG 1 , IMG 2 , ..., IMG j , ..., IMG Q },IMG j is the j-th sample image in IMG, the value of j ranges from 1 to Q, and Q is the number of sample images; S102, set variable r=1; S103, if r≤x, execute S104, otherwise, execute S110; x is the number of preset segmentation models; S104, set j=1; S105, if j≤Q, execute S106, execute S109; S106, to IMG j Identify the target object in the image and obtain multiple corresponding recognition boxes; S107, using the rth preset segmentation model to segment IMG j The target object identified in each corresponding recognition frame is segmented to obtain the corresponding segmented area; IMG j The corresponding segmentation area set A r j ={A r j1 , A r j2 , ..., A r jt , ..., A r jf(r,j) }, A r jt A r j The t-th segmentation region in , t ranges from 1 to f(r, j), f(r, j) is IMG j The number of corresponding segmented regions; S108, Get IMG j The corresponding ratio set K r j And get min(K r j ) as IMG j The corresponding minimum segmentation ratio; set j = j + 1, and execute S105; wherein, K r j ={K r j1 , K r j2 , ..., K r jt , ..., K r jf(r,j) }, K r jt K r j The tth proportion in K r jt =(SA r j / SF r j ), SA r j A r jt Corresponding size, SF r j A r jt The size of the corresponding recognition box; min() means taking the minimum value; S109, based on min(K r 1 ) to min(K r Q ) Get the target segmentation ratio KM corresponding to the rth preset segmentation model r ; Set r=r+1 and execute S103. S110, based on KM 1 To KM x Get k1.
3. The method according to claim 2, It is characterized in that KM r =(min(K r 1 )+min(K r 2 )+……+min(K r j )+……+min(K Q r )) / Q。 4. The method according to claim 2, It is characterized in that KM r It can be obtained through the following steps: S1, get the average value AK r and deviation value DK r , A.K. r =(min(K r 1 )+min(K r 2 )+……+min(K r j )+……+min(K r Q )) / Q, S2, traverse the data set (min(K r 1 ), min(K r 2 ), ..., min(K r j ), ..., min(K r Q )), if |min(K r j )-AK r ∣≤f*DK r , min(K r j ) is added as candidate data to the current candidate data set, the initial value of the current candidate data set is a null value, and f is a preset coefficient; S3, obtain the average value of all candidate data in the current candidate data set as KM r .
5. The method according to claim 2, It is characterized in that k1=(KM 1 +KM 2 +……+KM r +……+KM x ) / x。 6. The method according to claim 1, It is characterized in that The target objects include people and items carried by people.
7. An image processing device, It is characterized in that The device comprises: The recognition module is used to recognize the target object in the original image and obtain m recognition frames; the value of i ranges from 1 to m; A segmentation module is used to segment the target object identified in the i-th recognition frame to obtain a corresponding segmentation area; The first acquisition module is used to obtain i / SF i )>k1, the corresponding segmented area in the i-th recognition frame is used as the corresponding filling area AF i Otherwise, the area corresponding to the i-th recognition box is used as the corresponding filling area AF i ; Get the filling area AF corresponding to the m filling areas 1 To AF m Among them, SP i is the size of the segmented area in the i-th recognition frame, SF i is the size of the i-th recognition box; k1 is the first preset ratio threshold; The second acquisition module is used to acquire the overall filling area AF=(AF 1 ∪AF 2 ∪……∪AF i ……∪AF m ) as the target filling area of the original image; The processing module is used to fill the AF in the original image when it is determined that (SAF / S)≤k2, and obtain the original image after the filling process as the target image, SAF is the size corresponding to AF, S is the size of the original image, and k2 is the second preset ratio threshold.
8. An electronic device, It is characterized in that including a processor and a memory; The processor is used to execute the steps of the method according to any one of claims 1 to 6 by calling the program or instruction stored in the memory.
9. A non-transitory computer-readable storage medium, It is characterized in that The non-transitory computer-readable storage medium is used to store a program or instruction, and the program or instruction enables a computer to execute the steps of the method according to any one of claims 1 to 6.
Citation Information
Patent Citations
Indoor automatic drawing and modeling method based on planar image
CN110532602A
Moving target synchronous detection, identification and tracking method based on deep learning
CN111508006A
Image sample construction method, target recognition model training method and related device
CN116912621A
Target identification method of fisheye image, electronic equipment and storage medium
CN117765493A
Underwater weak supervision instance segmentation method based on bounding box labeling
CN118155056A
