A method, device and medium for eliminating specific targets in panoramic images

Abstract

The present invention discloses a method, device and medium for eliminating specific targets in a panoramic image, wherein the method includes: performing wide-angle image splitting on the panoramic image and optimizing the wide-angle deformation; performing target detection and identifying the visual features of the target object based on SAM for each frame of the panoramic image; performing continuous tracking of the target object based on an adaptive feature enhancement module AFE according to the visual features of the target object identified in each frame of the image, and determining the target object area to be eliminated in the panoramic image; performing image cropping on the target object area according to the determined target object area to be eliminated, and using ControlNet to effectively fill the cropped area based on the remaining background information. The present invention combines the SAM specific target recognition method and the background filling technology of ControlNet to improve the efficiency and quality of panoramic image processing, and promote the development of related industries to a higher level of intelligence and automation.

Description

Technical Field

[0001] The present invention relates to the technical field of eliminating specific targets in panoramic images, and in particular to a method, device and medium for eliminating specific targets in panoramic images. Background Art

[0002] Panoramic imaging technology has great application potential in many fields, such as monitoring systems, virtual reality (VR), urban planning, and film and television production, because it can provide a wide perspective and rich visual information. However, the widespread application of panoramic imaging also faces a significant challenge, that is, how to effectively and naturally remove specific interfering elements in the picture, such as pedestrians, vehicles, etc., while ensuring that the modified image maintains visual coherence and realism, and does not cause discomfort or confusion to the viewer.

[0003] Traditional methods often rely on manual editing or simple masking techniques. These methods are not only time-consuming and labor-intensive, but also difficult to maintain good results in complex dynamic scenes, especially when the target and background are highly integrated or there are complex interactions. With the rapid progress of deep learning technology in recent years, especially the application of unsupervised image segmentation and conditional image generation models, automated image processing, target removal and background filling have become possible. Nevertheless, the removal of specific targets in panoramic images still faces many technical difficulties, such as deformation processing caused by wide angles, stability of target tracking between consecutive frames, and natural completion of the background after removal.

[0004] Patent document CN108564527B discloses a method for panoramic content completion and repair based on a neural network. The invention scheme can perform image completion operations such as watermark removal and foreground occlusion removal based on panoramic data, overcoming the deformation and distortion problems of directly performing image completion on the two-dimensional storage form of the panorama in the related art, and significantly improving the display quality of panoramic content and the user experience in the content distribution process. However, it also has the following problems: 1. Its scheme is mainly aimed at static objects such as watermarks, and is not suitable for specific target elimination of panoramic images of dynamic interference elements; 2. The filling of the target cannot automatically guide the generation or modification of the image content based on the control signal (such as the segmentation mask). Summary of the invention

[0005] In response to the above problems, the purpose of the present invention is to provide a method, device and medium for eliminating specific targets in panoramic images, intelligently analyze the surrounding environment of the panoramic image, generate content that is highly consistent with the original background to fill the blank area left by the target elimination, and improve the efficiency and quality of panoramic image processing.

[0006] The embodiments of the present invention provide a method, a device and a medium for eliminating a specific target in a panoramic image.

[0007] A first aspect: A method for eliminating a specific target in a panoramic image, comprising:

[0008] S1, splitting the panoramic image into wide-angle images, and optimizing the wide-angle deformation of each frame image in the panoramic image;

[0009] S2, each frame of the panoramic image performs target detection based on SAM to identify the visual features of the target object;

[0010] S3, based on the visual features of the target object identified in each frame image, the target object is continuously tracked based on the adaptive feature enhancement module AFE to determine the target object area to be eliminated in the panoramic image;

[0011] S4. Based on the determined target object area to be eliminated, the target object area is cropped, and ControlNet is used to effectively fill the cropped area based on the remaining background information.

[0012] Furthermore, the step of splitting the panoramic image into wide-angle images and optimizing the wide-angle deformation in the panoramic image in S1 includes the following steps:

[0013] S11, mapping the panoramic image onto a three-dimensional spherical surface using spherical projection;

[0014] S12, splitting the single-frame panoramic image into 5 images at different positions according to the camera's field of view;

[0015] S13, then through the wide-angle image deformation correction optimization deep neural network, predict and correct the deformation caused by the wide-angle lens, merge the corrected images and restore them into an optimized panoramic picture after deformation correction. The process formula is:

[0016] (1)

[0017] (2)

[0018] (3)

[0019] in, It is a single image deformation correction module, which performs deformation correction on 5 images in different positions. Assemble and restore modules for storyboards. Will The 5 corrected results at different positions are combined to restore one picture. Formula 1 and 2 are combined to form Formula 3. For the deformation correction network, is a collection of panoramic images, A collection of images of optimized, warped panoramas.

[0020] Furthermore, the target objects in S2 include people, animals and vehicles.

[0021] Furthermore, in S2, each frame of the panoramic image performs target detection based on SAM to identify visual features of the target object, including the steps of:

[0022] S21. Detect the target object in the optimized panoramic image based on SAM and extract the common visual features of the target object. The process formula is:

[0023] (4)

[0024] Among them, SAM is the Segment Anything model. To optimize the image collection for panoramas, The boundary detection result extracted by the SAM model is from the panoramic picture N image regions with clear boundaries are extracted;

[0025] S22. Through transfer learning, fine-tune the performance of the SAM model on human, animal and vehicle target objects, and obtain the Mask features of human, animal and vehicle target objects. The process formula is:

[0026] (5)

[0027] (6)

[0028] (7)

[0029] in, It is a SAM model fine-tuned with human samples. Extract the Mask feature of the person detection area , is a SAM model fine-tuned with automobile samples. For this model in a single image The Mask features of the car detection area are extracted. This is a SAM model that has been fine-tuned with animal samples. For this model in a single image The Mask features of the animal detection area are extracted from the above;

[0030] S23, using the Mask features of the target objects of people, animals and vehicles extracted in S22 as the strong feature prompt of the SAM model, extracting the enhanced visual features of the target objects, the process formula is:

[0031] (8)

[0032] Among them, prompt means that As the input part of the prompt enhancement module that comes with the SAM model, Extract enhanced visual features of the target object for the re-adapted SAM detection results after the enhanced input, is the object label corresponding to each detection result (such as person, car, animal).

[0033] The SAM detection results that are re-adapted after the enhanced input not only improve the recognition accuracy of types such as people, cars, and animals, but also can obtain more accurate boundary results for the overall boundary recognition of various other objects.

[0034] Furthermore, in S3, based on the visual features of the target object identified in each frame image, the target object is continuously tracked based on the adaptive feature enhancement module AFE to determine the target object area to be eliminated in the panoramic image, including the steps of:

[0035] S31, introduce the adaptive feature enhancement module AFE, which converts the target object identified in the previous frame into a specific information Mask feature matrix that can be used by the SAM model , the process formula is:

[0036] (9)

[0037] in, It represents the SAM detection result of the previous frame image after the enhanced input is re-adapted. is the object label corresponding to each detection result in the previous frame, and AFE is the feature enhancement module for adaptive adjustment of consecutive frames;

[0038] S32, the specific information Mask feature matrix of the above frame , as a strong feature prompt of the SAM model, extract the optimized visual features of the target object. The process formula is:

[0039] (10)

[0040] in, Indicates that As the input part of the prompt enhancement module that comes with the SAM model, Indicates that the Mask feature matrix of the previous frame specific information , as a strong feature prompt of the SAM model;

[0041] S33, repeat S31 to S32, extract N frames of images in the panoramic image in turn, and identify all target object areas to be eliminated:

[0042] (11)

[0043] Among them, Region is the area to be eliminated. To detect the boundary information of the target, Label information for the detected target.

[0044] Therefore, each frame of the panoramic image can be enhanced based on the current frame information. And the enhancement information of the previous frame Under the joint guidance, accurate and complete target detection results are obtained: is the target detection result of all frames, is the corresponding category label.

[0045] Furthermore, the specific information includes lighting, angle and background complexity.

[0046] Furthermore, in S4, according to the determined target object area to be eliminated, the target object area is cropped, and the cropped area is effectively filled based on the remaining background information using ControlNet, including:

[0047] ControlNet is used to accept the target object to be eliminated area provided by SAM as the control signal, and the missing background content is synthesized in the to-be-eliminated area based on the remaining background information. The process formula is: (12)

[0048] in, After erasing the target, refill the generated new complete image. Fill the generated module for the image, To detect the boundary information of the target.

[0049] Furthermore, when synthesizing the missing background content in the area to be eliminated based on the remaining background information using ControlNet, a spatiotemporal consistency module is introduced to maintain the smooth transition and lighting consistency of the background in the dynamic scene of the panoramic image. The process formula is: (13)

[0050] in, The image after background completion of the previous frame.

[0051] A second aspect: An electronic device comprises a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the method provided in the first aspect when executing the program.

[0052] A third aspect: A non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the method provided in the first aspect.

[0053] Beneficial effects of the present invention:

[0054] The method for eliminating specific targets in panoramic images of the present invention combines the SegmentAnything (SAM) specific target recognition method and the background filling technology of ControlNet. By using SAM, objects of different categories in the panoramic image can be accurately segmented and recognized, and a high recognition accuracy rate is maintained. It makes full use of large-scale context information and self-attention mechanism, and effectively distinguishes the target objects without manual labeling. In the present invention, ControlNet is responsible for intelligently analyzing the surrounding environment after SAM recognizes and marks the specific target, and generating content that is highly consistent with the original background to fill the blank area left by the target elimination. The entire model has a detailed understanding of background texture, lighting and color, and also achieves consistency in the time dimension, ensuring a natural transition between consecutive frames after the target is eliminated in the video sequence, avoiding abrupt visual faults. It realizes full-process automated processing from target recognition, precise elimination and background completion, and is suitable for processing panoramic images containing a large number of dynamic interference elements, bringing efficiency and quality improvements to panoramic image processing, and promoting related industries to a higher level of intelligence and automation. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] Figure 1 A schematic diagram of a flow chart of a method for eliminating specific targets in a panoramic image according to the present invention;

[0056] Figure 2 A schematic diagram of the process structure of a device for removing specific targets in a panoramic image according to the present invention;

[0057] Figure 3 It is a structural schematic diagram of the electronic device of the present invention; DETAILED DESCRIPTION

[0058] Embodiments of the present invention are described in detail below, examples of which are shown in the accompanying drawings, wherein the same or similar symbols throughout represent the same or similar elements or elements with the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and cannot be understood as limiting the present invention.

[0059] At present, there are still many technical limitations in the field of eliminating specific targets in panoramic images. These problems directly restrict the image quality and application effect, which are specifically reflected in the following aspects:

[0060] 1. Difficulty in processing wide-angle deformation: Panoramic images usually involve extremely wide viewing angles. The resulting barrel-shaped or pincushion-shaped deformations pose a huge challenge to target recognition and tracking. Traditional image processing models are often designed based on images taken with regular lenses. When directly applied to panoramic images, they are prone to reduced recognition accuracy, especially at the edges of objects and in areas with distorted shapes. Therefore, how to accurately identify and track specific targets in panoramic images with severe deformation has become a key issue that needs to be solved urgently.

[0061] 2. Generalization and adaptability of target recognition: The lighting conditions in different scenes, the diversity of targets, and the characteristics of different panoramic imaging devices all put the generalization ability and adaptability of the algorithm to the test. Existing technologies often perform well under specific information, but are difficult to be universally applied to various complex practical application scenarios and lack sufficient flexibility and universality.

[0062] 3. Continuous tracking stability in dynamic scenes: In dynamically changing panoramic images, the position, posture and occlusion of the target object will change over time, which places extremely high demands on the stability of target tracking. Some existing tracking technologies are prone to losing the target or mismatching when faced with fast motion, frequent occlusion or confusion of similar targets, thus affecting the subsequent elimination and completion effects. Therefore, how to achieve high-precision and robust target tracking in complex dynamic environments is the key to improving the coherence and realism of panoramic image processing.

[0063] 4. Difficulty of natural background completion: The blank area left after the specific target is eliminated needs to be accurately and naturally integrated with the surrounding environment, which requires the completion algorithm to not only understand the texture, color and structure of the background, but also take into account the changes in light and shadow and the spatial perspective relationship to achieve a seamless visual effect. Especially for panoramic images with complex backgrounds, how to generate completion content that conforms to physical laws and is consistent with the context is a major technical challenge.

[0064] In view of the above problems, the present invention provides a method for eliminating specific targets in panoramic images. By integrating the advanced technologies of SAM and ControlNet, a comprehensive solution is proposed, which aims to effectively solve the key problems such as deformation processing, generalization adaptability, continuous tracking, background completion and so on in the process of eliminating specific targets in panoramic images. It can efficiently identify specific targets such as people, animals, and cars in panoramic images, improve the effect and quality of panoramic image processing, and meet more diversified and high-quality application needs.

[0065] As an advanced unsupervised target recognition technology, SAM can accurately segment and identify objects of different categories in panoramic images, and maintain a high recognition accuracy even when the target and the background have similar colors and textures. Its core advantage lies in the use of a wide range of contextual information and self-attention mechanism, which can effectively distinguish target objects without manual labeling.

[0066] ControlNet is an image processing model based on conditional generation, which can guide the generation or modification of image content according to input control signals (such as segmentation masks). In the present invention, ControlNet plays a key role. After SAM identifies and marks a specific target, it intelligently analyzes the surrounding environment and generates content that is highly consistent with the original background to fill the blank area left by the target elimination. This requires not only that the model has a detailed understanding of background texture, lighting, and color, but also that it must achieve consistency in the time dimension to ensure a natural transition between consecutive frames after the target is eliminated in the video sequence to avoid abrupt visual discontinuities.

[0067] Figure 1 A schematic flow chart of a method for eliminating specific targets in a panoramic image provided by an embodiment of the present invention, the method comprising:

[0068] S1. splitting the panoramic image into wide-angle images, and optimizing the wide-angle deformation of each frame image in the panoramic image.

[0069] First, an improved preprocessing step is adopted, which combines spherical projection technology and deep learning models to achieve effective rectification of the original panoramic images, such as Figure 2 As shown in S1, the panoramic image is mapped onto a three-dimensional sphere using a spherical projection method, and a single-frame panoramic image is split into five images at different positions according to the camera's field of view. Subsequently, each panoramic image is converted into a deformation-corrected image through a deep neural network optimized for wide-angle image correction, and then the corrected images are merged and restored into an optimized panoramic image after deformation correction.

[0070] The process formula is:

[0071] (1)

[0072] (2)

[0073] (3)

[0074] in, It is a single image deformation correction module, which performs deformation correction on 5 images at different positions (Formula 1). Assemble and restore modules for storyboards. Will The corrected results of the five different positions are combined to restore one picture (Formula 2). Formulas 1 and 2 can be combined to form Formula 3. For the deformation correction network, is a collection of panoramic images, A set of images of the panoramic image after deformation correction.

[0075] The deformation correction network used A large number of deformation samples are used for learning during the training process, which can predict and correct the barrel or pincushion deformation caused by wide-angle lenses.

[0076] Deformation Correction Network Convolutional neural networks are used to capture local features and combine them with global context information to ensure that image details and structural integrity are retained while correcting deformation. This not only improves the accuracy of target recognition, but also lays the foundation for subsequent tracking and completion operations, ensuring that deformation no longer becomes an obstacle to recognition and tracking throughout the entire processing flow.

[0077] S2: Target detection is performed on each frame of the panoramic image based on SAM to identify the visual features of the target object.

[0078] In order to solve the generalization and adaptability problems of target recognition, the unsupervised edge recognition ability of the SAM model with strong generalization is combined with the accurate recognition ability of characteristic target objects. Based on SAM, target object detection is performed on the optimized panoramic pictures to extract the common visual features of target objects. The process formula is:

[0079] (4)

[0080] Among them, SAM is the Segment Anything model. A collection of images optimized for panoramas. The boundary detection result extracted by the SAM model is from the image N image regions with clear boundaries are extracted from A collection of images optimized for panoramas.

[0081] The strong generalization edge detection capability of the SAM model can enhance the performance of the specific target object recognition model in different scenarios. The SAM model itself has undergone large-scale unsupervised pre-training and uses a panoramic image dataset containing rich scenes and target types to extract common visual features.

[0082] In addition, the performance of the SAM model is further fine-tuned on specific target objects through transfer learning techniques, such as Figure 2As shown in the figure, for commonly used elimination objects such as people, animals, and vehicles, the extraction results of the person recognition and vehicle recognition models are combined to ensure the best recognition effect in specific applications, improve the generalization ability of the algorithm, fine-tune the performance of the SAM model on the target objects of people, animals, and vehicles, and obtain the Mask features of the target objects of people, animals, and vehicles. The process formula is:

[0083] (5)

[0084] (6)

[0085] (7)

[0086] in, It is a SAM model fine-tuned with human samples. Extract the human detection area , is a SAM model fine-tuned with automobile samples. For this model in a single image The automobile detection area is extracted from the above. This is a SAM model that has been fine-tuned with animal samples. For this model in a single image The animal detection area is obtained by extraction.

[0087] Then, the Mask features of target objects such as people, animals and vehicles are used as the strong feature prompt of the SAM model to extract the enhanced visual features of the target objects. The process formula is:

[0088] (8)

[0089] Among them, prompt means that As the input part of the prompt enhancement module that comes with the SAM model, To enhance the SAM detection results after input refitting, is the object label corresponding to each detection result (such as person, car, animal).

[0090] The SAM detection results that are re-adapted after the enhanced input not only improve the recognition accuracy of types such as people, cars, and animals, but also can obtain more accurate boundary results for the overall boundary recognition of various other objects.

[0091] In addition, the performance of the SAM model in other specific fields can be further fine-tuned through transfer learning technology to ensure the best recognition effect in specific applications and improve the generalization ability of the algorithm.

[0092] S3, based on the visual features of the target object identified in each frame image, the target object is continuously tracked based on the adaptive feature enhancement module AFE to determine the target object area to be eliminated in the panoramic image;

[0093] In order to achieve continuous and stable tracking of specific target objects when processing dynamic panoramic images, a dynamic target object tracking framework is established. Within the dynamic target object tracking framework, the framework is not only based on the SAM recognition result of the current frame, but also integrates the information of historical frames, and uses the long short-term memory network to capture the motion trend and context dependency of the target.

[0094] The dynamic target object tracking framework iteratively updates the state (position, size, speed, etc.) of the target object in each frame to enhance the confirmation of the target object's identity and reduce misidentification caused by background changes or occlusions.

[0095] In order to better guide the SAM model by using the recognition results of the specific target object in the previous frame, an adaptive feature enhancement module (AFE) is introduced. This module can convert the target object recognized in the previous frame into a specific information Mask feature matrix that can be used by the SAM model. , the process formula is:

[0096] (9)

[0097] in, It represents the SAM detection result of the previous frame image after the enhanced input is re-adapted. is the object label corresponding to each detection result in the previous frame, Feature enhancement module that adaptively adjusts for consecutive frames.

[0098] Mask feature matrix with specific information (such as lighting, angle, background complexity, etc.) , as a strong feature prompt of the SAM model, dynamically adjusts the feature extraction strategy based on the specific information of the input image, thereby improving the adaptability of the model to changing conditions and extracting the optimized visual features of the target object. The process formula is:

[0099] (10)

[0100] in, Similar to S23, it means As the input part of the prompt enhancement module that comes with the SAM model, Indicates that the specific information Mask feature matrix of the previous frame , as a strong feature prompt of the SAM model.

[0101] The AFE module uses the attention mechanism to focus on the most discriminative areas in the image, and combines contextual information to enable SAM to refer to the location of the target object in the previous frame when facing a new image, thereby identifying the target more efficiently.

[0102] Through this strategy, the dynamic target tracking framework significantly improves the stability and robustness of tracking, and also improves the consistency of target recognition in consecutive frames of the video, providing a solid foundation for subsequent elimination and completion.

[0103] Repeat N times, extract N frames of images in the panoramic image in turn, and identify all target object areas to be eliminated:

[0104] (11)

[0105] Among them, Region is the area to be eliminated, and objects are eliminated and background is filled in the next step. To detect the boundary information of the target, It is the category information of the detected target.

[0106] Therefore, each frame of the panoramic image can be enhanced based on the current frame information. And the enhancement information of the previous frame Under the joint guidance, accurate and complete target detection results are obtained: is the target detection result of all frames, is the corresponding category label.

[0107] S4. According to the determined target object area to be eliminated, the target object area is cropped, and ControlNet is used to effectively fill the cropped area based on the remaining background information.

[0108] In order to solve the problem of background completion, the present invention fully exploits the potential of ControlNet and realizes natural completion after eliminating specific targets through background generation and fusion technology driven by deep learning.

[0109] ControlNet accepts the target object to be eliminated area provided by SAM as the control signal, and uses the mask-guided conditional generation network to synthesize the missing background content. The ControlNet network learns a large amount of real scenes and synthetic data during the training phase to understand the complex background texture, lighting and structure. By introducing style transfer technology in the completion process, the generated background is not only coordinated with the surrounding environment in texture, such as Figure 2 As shown in S4, the color and style can be kept consistent, further improving the naturalness of vision. The missing background content can be synthesized in the area to be eliminated based on the remaining background information. The process formula is:

[0110] (12)

[0111] in, After erasing the target, refill the image to generate a new complete image. Fill the generated module for the image, To detect the boundary information of the target.

[0112] In order to enhance the realism of the completed content, a spatiotemporal consistency module is further introduced. This module considers the coherence of adjacent frames during the completion process, that is, the image after background completion of the previous frame is passed into the generation model as guidance information to ensure the smooth transition and lighting consistency of the background in dynamic scenes. The process formula is:

[0113] (13)

[0114] in, The image after background completion of the previous frame.

[0115] The present invention also discloses a device for eliminating specific targets in a panoramic image used in the above method, such as Figure 2 As shown, including:

[0116] The wide-angle deformation processing module is used to split the acquired panoramic image into wide-angle images and optimize the wide-angle deformation of each frame image in the panoramic image.

[0117] The target detection module is used to perform target detection on each frame of the panoramic image based on SAM and identify the visual features of the target object.

[0118] The continuous target tracking and edge determination module is used to continuously track the target object based on the visual features of the target object identified in each frame image and based on the adaptive feature enhancement module AFE, and determine the area of ​​the target object to be eliminated in the panoramic image.

[0119] The target elimination and background filling module is used to crop the target object area according to the determined target object area to be eliminated, and use ControlNet to effectively fill the cropped area based on the remaining background information.

[0120] The present invention combines panoramic images with specific target recognition SegmentAnything (SAM) and ControlNet's background filling technology to achieve full-process automated processing from target recognition, precise elimination and background completion. It is particularly suitable for processing panoramic images containing a large number of dynamic interference elements, which improves the efficiency and quality of panoramic image processing and promotes related industries to develop towards a higher level of intelligence and automation.

[0121] The present invention also provides an electronic device, Figure 3 A schematic diagram of the structure of an electronic device provided by an embodiment of the present invention, such as Figure 3 As shown, the electronic device may include: a processor, a communications interface, a memory, and a communication bus, wherein the processor, the communications interface, and the memory communicate with each other via the communication bus. The processor may call the logic instructions in the memory, for example, to execute the following method:

[0122] S1, splitting the panoramic image into wide-angle images, and optimizing the wide-angle deformation of each frame image in the panoramic image;

[0123] S2, each frame of the panoramic image performs target detection based on SAM to identify the visual features of the target object;

[0124] S3, based on the visual features of the target object identified in each frame image, the target object is continuously tracked based on the adaptive feature enhancement module AFE to determine the target object area to be eliminated in the panoramic image;

[0125] S4. Based on the determined target object area to be eliminated, the target object area is cropped, and ControlNet is used to effectively fill the cropped area based on the remaining background information.

[0126] In addition, the logic instructions in the above-mentioned memory can be implemented in the form of software functional units and can be stored in a computer-readable storage medium when sold or used as an independent product. Based on this understanding, the technical solution of the present invention can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium, including several instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.

[0127] An embodiment of the present invention further provides a non-transitory computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the method provided in each of the above embodiments is implemented, for example, including:

[0128] S1, splitting the panoramic image into wide-angle images, and optimizing the wide-angle deformation of each frame image in the panoramic image;

[0129] S2, each frame of the panoramic image performs target detection based on SAM to identify the visual features of the target object;

[0130] S3, based on the visual features of the target object identified in each frame image, the target object is continuously tracked based on the adaptive feature enhancement module AFE to determine the target object area to be eliminated in the panoramic image;

[0131] S4. Based on the determined target object area to be eliminated, the target object area is cropped, and ControlNet is used to effectively fill the cropped area based on the remaining background information.

[0132] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0133] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0134] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for eliminating specific targets in a panoramic image, characterized in that: include: S1, splitting the panoramic image into wide-angle images, and optimizing the wide-angle deformation of each frame image in the panoramic image; S2, each frame of the panoramic image performs target detection based on SAM to identify the visual features of the target object; S3, based on the visual features of the target object identified in each frame image, the target object is continuously tracked based on the adaptive feature enhancement module AFE to determine the target object area to be eliminated in the panoramic image; S4, according to the determined target object area to be eliminated, the target object area is cropped, and ControlNet is used to effectively fill the cropped area based on the remaining background information; In S1, the panoramic image is split into wide-angle images, and the wide-angle deformation in the panoramic image is optimized, including the steps of: S11, mapping the panoramic image onto a three-dimensional spherical surface using spherical projection; S12, splitting the single-frame panoramic image into 5 images at different positions according to the camera's field of view; S13, then through the wide-angle image deformation correction optimization deep neural network, predict and correct the deformation caused by the wide-angle lens, merge the corrected images and restore them into an optimized panoramic picture after deformation correction. The process formula is: （1） （2） （3） in, It is a single image deformation correction module, which performs deformation correction on 5 images in different positions. Assemble and restore modules for storyboards. Will The 5 corrected results at different positions are combined to restore one picture. Formula 1 and 2 are combined to form Formula 3. For the deformation correction network, is a collection of panoramic images, A collection of images of optimized panoramic images after deformation correction; The target objects in S2 include people, animals and vehicles; In the S2, each frame of the panoramic image performs target detection based on SAM to identify the visual features of the target object, including the following steps: S21. Detect the target object in the optimized panoramic image based on SAM and extract the common visual features of the target object. The process formula is: （4） Among them, SAM is the Segment Anything model. To optimize the image collection for panoramas, The boundary detection result extracted by the SAM model is from the panoramic picture N image regions with clear boundaries are extracted; S22. Through transfer learning, fine-tune the performance of the SAM model on human, animal and vehicle target objects, and obtain the Mask features of human, animal and vehicle target objects. The process formula is: （5） （6） （7） in, The SAM model is fine-tuned with human samples. Extract the Mask feature of the person detection area , is a SAM model fine-tuned with automobile samples. For this model in a single image The Mask features of the car detection area are extracted. This is a SAM model that has been fine-tuned with animal samples. For this model in a single image The Mask features of the animal detection area are extracted from the above; S23, using the Mask features of the target objects of people, animals and vehicles extracted in S22 as the strong feature prompt of the SAM model, extracting the enhanced visual features of the target objects, the process formula is: （8） Among them, prompt means that As the input part of the prompt enhancement module that comes with the SAM model, Extract enhanced visual features of the target object for the re-adapted SAM detection results after the enhanced input, is the object label corresponding to each detection result.

2. The method for eliminating specific targets in a panoramic image according to claim 1, characterized in that: In S3, based on the visual features of the target object identified in each frame image, the target object is continuously tracked based on the adaptive feature enhancement module AFE to determine the target object area to be eliminated in the panoramic image, including the steps of: S31, Introduction The adaptive feature enhancement module AFE converts the target object identified in the previous frame into a specific information Mask feature matrix that can be used by the SAM model , the process formula is: （9） in, It represents the SAM detection result of the previous frame image after the enhanced input is re-adapted. is the object label corresponding to each detection result in the previous frame, and AFE is the feature enhancement module for adaptive adjustment of consecutive frames; S32, the specific information Mask feature matrix of the above frame , as a strong feature prompt of the SAM model, extract the optimized visual features of the target object. The process formula is: （10） in, Indicates that As the input part of the prompt enhancement module that comes with the SAM model, Indicates that the Mask feature matrix of the previous frame specific information , as a strong feature prompt of the SAM model; S33, repeat S31 to S32, extract N frames of images in the panoramic image in turn, and identify all target object areas to be eliminated: （11） Among them, Region is the area to be eliminated. To detect the boundary information of the target, Label information for the detected target.

3. The method for eliminating specific targets in a panoramic image according to claim 2, characterized in that: The specific information includes lighting, angle and background complexity.

4. The method for eliminating specific targets in a panoramic image according to claim 2, characterized in that: In S4, according to the determined target object area to be eliminated, the target object area is cropped, and the cropped area is effectively filled based on the remaining background information using ControlNet, including: ControlNet is used to accept the target object to be eliminated area provided by SAM as the control signal, and the missing background content is synthesized in the to-be-eliminated area based on the remaining background information. The process formula is: （12） in, After erasing the target, refill the generated new complete image. Fill the generated module for the image, To detect the boundary information of the target.

5. The method for eliminating specific targets in a panoramic image according to claim 4, characterized in that: When using ControlNet to synthesize the missing background content in the area to be eliminated based on the remaining background information, the spatiotemporal consistency module is introduced to maintain the smooth transition and lighting consistency of the background in the dynamic scene of the panoramic image. The process formula is: （13） in, The image after background completion of the previous frame.

6. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the steps of a method for eliminating a specific target in a panoramic image as described in any one of claims 1 to 5 are implemented.

7. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of a method for eliminating a specific target in a panoramic image as described in any one of claims 1 to 5 are implemented.

Images