Foreground target extraction method and device in three-dimensional scene, equipment and storage medium

Abstract

The embodiment of the invention provides a foreground target extraction method and device in a three-dimensional scene, equipment and a storage medium, and relates to the technical field of three-dimensional image processing. The method comprises the following steps: performing three-dimensional reconstruction on a target scene based on a scene image to obtain a three-dimensional Gaussian point cloud; obtaining a segmentation parameter in response to a target selection instruction corresponding to one scene image, and inputting the segmentation parameter and the segmented scene image into a target segmentation model to obtain a foreground target mask; projecting the three-dimensional Gaussian point cloud on an imaging plane corresponding to the segmented scene image to obtain a projection point corresponding to each Gaussian point in the three-dimensional Gaussian point cloud, and selecting part of Gaussian points as foreground Gaussian points based on the projection positions of the projection points and the foreground target mask; and obtaining a three-dimensional foreground target based on the foreground Gaussian point. According to the method, a mapping bridge between a two-dimensional scene image and a three-dimensional Gaussian point cloud is constructed, and the three-dimensional Gaussian point cloud is constrained by using a two-dimensional foreground target mask, so that a three-dimensional foreground target is accurately separated, and the separation and extraction efficiency of the foreground target is improved.

Description

Technical Field

[0001] The present application relates to the technical field of three-dimensional image processing, and in particular to a method, device, equipment and storage medium for extracting a foreground object in a three-dimensional scene. Background Art

[0002] In the process of 3D scene modeling, it is widely used to separate the foreground and background of the 3D model obtained by modeling. This separation can separate the object of interest as the foreground from the background, which is helpful for subsequent rendering and analysis.

[0003] In the related art, when separating foreground and background, foreground targets are extracted based on point cloud data. By identifying and separating foreground targets, foreground and background are effectively distinguished from the three-dimensional model. However, when extracting foreground targets, this method requires that the foreground targets be determined before modeling. This means that every time a new foreground target is replaced, the entire modeling process needs to be repeated, including steps such as data collection, processing, and model building. This process is time-consuming, resulting in low processing efficiency for foreground and background separation. Summary of the invention

[0004] The main purpose of the embodiments of the present application is to propose a method, device, equipment and storage medium for extracting foreground objects in a three-dimensional scene, so as to improve the processing efficiency of extracting foreground objects in a three-dimensional model.

[0005] To achieve the above object, a first aspect of an embodiment of the present application proposes a method for extracting a foreground object in a three-dimensional scene, comprising:

[0006] Acquire scene images of different perspectives acquired by a plurality of image acquisition devices pre-deployed in the target scene, and perform three-dimensional reconstruction of the target scene based on the scene images to obtain a three-dimensional Gaussian point cloud;

[0007] In response to a target selection instruction corresponding to one of the scene images as a segmented scene image, a segmentation parameter is obtained, and the segmentation parameter and the segmented scene image are input into a target segmentation model for image segmentation to obtain a foreground target mask;

[0008] Projecting the three-dimensional Gaussian point cloud onto an imaging plane corresponding to the segmented scene image to obtain projection points corresponding to each Gaussian point in the three-dimensional Gaussian point cloud, and selecting part of the Gaussian points as foreground Gaussian points based on the projection positions of the projection points and the foreground target mask;

[0009] A three-dimensional foreground object is obtained based on the foreground Gaussian points.

[0010] In some embodiments, obtaining a three-dimensional foreground object based on the foreground Gaussian points includes:

[0011] Perform at least one iteration process, in each iteration, select the scene image corresponding to other perspectives as the segmented scene image, use the foreground Gaussian point of the previous iteration as the three-dimensional Gaussian point cloud, and update the foreground Gaussian point;

[0012] A three-dimensional foreground object is obtained based on the foreground Gaussian points obtained in the last iteration.

[0013] In some embodiments, obtaining the segmentation parameter in response to the target selection instruction corresponding to one of the scene images as the segmented scene image includes:

[0014] If one of the scene images is selected as the segmented scene image, obtaining a target selection instruction corresponding to the segmented scene image;

[0015] Obtaining the selected position coordinates based on the target selection instruction, and obtaining the segmentation granularity parameters corresponding to the selected position coordinates;

[0016] The selected position coordinates and the segmentation granularity parameters are used as the segmentation parameters.

[0017] In some embodiments, inputting the segmentation parameters and the segmented scene image into a target segmentation model to perform image segmentation to obtain a foreground target mask includes:

[0018] Inputting the segmentation parameters and the segmented scene image into a target segmentation model to perform image segmentation to obtain an initial foreground image;

[0019] If an adjustment selection instruction for the initial foreground image is responded to, an updated segmentation parameter is generated, and the updated segmentation parameter and the segmented scene image are input into the target segmentation model for image segmentation to obtain an updated initial foreground image;

[0020] If a stop segmentation instruction for the initial foreground image is responded to, the foreground object mask is generated according to the initial foreground image; otherwise, the updated segmentation parameters are continuously acquired until a stop segmentation instruction is responded to.

[0021] In some embodiments, if responding to an adjustment selection instruction for the initial foreground image, generating updated segmentation parameters includes:

[0022] Acquire the adjustment position coordinates of the adjustment selection instruction, and adjust the segmentation granularity parameter according to the positional relationship between the adjustment position coordinates and the initial foreground image to obtain an updated granularity parameter;

[0023] The selected position coordinates and the update granularity parameter are used as the update segmentation parameter.

[0024] In some embodiments, adjusting the segmentation granularity parameter according to the positional relationship between the adjusted position coordinates and the initial foreground image to obtain an updated granularity parameter includes:

[0025] If the adjusted position coordinates are outside the initial foreground image, increasing the segmentation granularity parameter;

[0026] If the adjusted position coordinates are within the initial foreground image, the segmentation granularity parameter is reduced.

[0027] In some embodiments, generating the foreground object mask according to the initial foreground image includes:

[0028] Binarizing the initial foreground image to obtain an initial foreground mask;

[0029] A preset expansion rate is obtained, and the initial foreground mask is expanded based on the preset expansion rate to obtain the foreground object mask.

[0030] In some embodiments, projecting the three-dimensional Gaussian point cloud onto an imaging plane corresponding to the segmented scene image to obtain a projection point corresponding to each Gaussian point in the three-dimensional Gaussian point cloud includes:

[0031] Acquire an intrinsic parameter matrix and an extrinsic parameter matrix of an image acquisition device corresponding to the segmented scene image, wherein the extrinsic parameter matrix includes a rotation matrix and a translation vector;

[0032] Acquire the spatial coordinate value of the Gaussian point, subtract the translation vector from the spatial coordinate value to obtain a first intermediate value, and calculate the product of the first intermediate value and the rotation matrix to obtain a second intermediate value;

[0033] The projection position of the Gaussian point on the imaging plane is obtained based on the second intermediate value, the intrinsic parameter matrix and the translation vector, and the projection point is determined based on the projection position.

[0034] In some embodiments, the selecting part of the Gaussian points as foreground Gaussian points based on the projection positions of the projection points and the foreground object mask includes:

[0035] Obtaining a foreground area of ​​the segmented scene image according to the foreground object mask;

[0036] When the projection position of the projection point is located in the foreground area, a Gaussian point corresponding to the projection point is selected as the foreground Gaussian point.

[0037] To achieve the above-mentioned purpose, a second aspect of an embodiment of the present application provides a device for extracting a foreground object in a three-dimensional scene, comprising:

[0038] A three-dimensional reconstruction module is used to obtain scene images of different perspectives acquired by multiple image acquisition devices pre-deployed in the target scene, and to perform three-dimensional reconstruction of the target scene based on the scene images to obtain a three-dimensional Gaussian point cloud;

[0039] A two-dimensional segmentation module: used for obtaining segmentation parameters in response to a target selection instruction corresponding to one of the scene images as a segmented scene image, inputting the segmentation parameters and the segmented scene image into a target segmentation model for image segmentation, and obtaining a foreground target mask;

[0040] A mask filtering module is used to project the three-dimensional Gaussian point cloud onto an imaging plane corresponding to the segmented scene image to obtain projection points corresponding to each Gaussian point in the three-dimensional Gaussian point cloud, and select part of the Gaussian points as foreground Gaussian points based on the projection positions of the projection points and the foreground target mask;

[0041] Target generation module: used to obtain a three-dimensional foreground target based on the foreground Gaussian points.

[0042] To achieve the above objectives, a third aspect of an embodiment of the present application proposes an electronic device, which includes a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the method described in the first aspect is implemented.

[0043] To achieve the above-mentioned purpose, the fourth aspect of an embodiment of the present application proposes a storage medium, which is a storage medium. The storage medium stores a computer program, and when the computer program is executed by a processor, the method described in the first aspect is implemented.

[0044] The foreground target extraction method, device, equipment and storage medium in a three-dimensional scene proposed in the embodiment of the present application obtains scene images of different perspectives obtained by multiple image acquisition devices pre-deployed in the target scene, and performs three-dimensional reconstruction of the target scene based on the scene images to obtain a three-dimensional Gaussian point cloud; obtains segmentation parameters in response to a target selection instruction corresponding to one of the scene images as a segmented scene image, inputs the segmentation parameters and the segmented scene image into the target segmentation model for image segmentation, and obtains a foreground target mask; projects the three-dimensional Gaussian point cloud onto the imaging plane corresponding to the segmented scene image to obtain the projection points corresponding to each Gaussian point in the three-dimensional Gaussian point cloud, selects some Gaussian points as foreground Gaussian points based on the projection positions of the projection points and the foreground target mask; and obtains a three-dimensional foreground target based on the foreground Gaussian points. The embodiment of the present application generates a corresponding foreground target mask using the segmentation parameters for a certain perspective obtained in the interactive process, then projects the three-dimensional Gaussian point cloud onto the imaging plane of the perspective, and selects foreground Gaussian points based on the foreground target mask to obtain a three-dimensional foreground target. This method builds a mapping bridge between the two-dimensional scene image and the three-dimensional Gaussian point cloud, and uses the two-dimensional foreground target mask to constrain the three-dimensional Gaussian point cloud. There is no need to perform the three-dimensional reconstruction process when selecting different foreground targets, nor is there a need to select the foreground target before three-dimensional reconstruction. The segmentation parameters are obtained through the interactive process, and the mask corresponding to the foreground target is obtained based on the segmentation parameters. The mask is then used to screen the three-dimensional Gaussian point cloud, which can accurately separate the three-dimensional foreground target and improve the separation and extraction efficiency of the foreground target. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] Figure 1 It is a flowchart of a method for extracting foreground objects in a three-dimensional scene provided in an embodiment of the present application.

[0046] Figure 2 It is a schematic diagram of an image acquisition device for three-dimensional reconstruction provided in an embodiment of the present application.

[0047] Figure 3 It is a schematic diagram of an implementation scenario provided in an embodiment of the present application.

[0048] Figure 4 It is a flowchart of obtaining segmentation parameters in response to a target selection instruction corresponding to one of the scene images as a segmented scene image provided by an embodiment of the present application.

[0049] Figure 5 It is a flowchart provided by an embodiment of the present application for simultaneously inputting segmentation parameters and a segmented scene image into a target segmentation model to perform image segmentation and obtain a foreground target mask.

[0050] Figure 6 It is a schematic diagram of generating updated segmentation parameters according to an adjustment selection instruction provided in an embodiment of the present application.

[0051] Figure 7This is a flowchart of generating a foreground target mask according to an initial foreground image provided by an embodiment of the present application.

[0052] Figure 8 It is a flowchart provided in an embodiment of the present application for projecting a three-dimensional Gaussian point cloud onto an imaging plane corresponding to a segmented scene image to obtain a projection point corresponding to each Gaussian point in the three-dimensional Gaussian point cloud.

[0053] Fig. 9 It is a schematic diagram of the projection points provided in the embodiment of the present application.

[0054] Fig.10 A schematic diagram of the mask filtering process of foreground Gaussian points provided in an embodiment of the present application.

[0055] Fig.11 It is a flow chart of obtaining a three-dimensional foreground target based on foreground Gaussian points provided in an embodiment of the present application.

[0056] Fig.12 A schematic diagram of the iterative process of updating foreground Gaussian points provided in an embodiment of the present application.

[0057] Fig.13 It is an overall flow chart of the method for extracting foreground objects in a three-dimensional scene provided in an embodiment of the present application.

[0058] Fig.14 It is a schematic diagram of a foreground object extraction method in a three-dimensional scene provided in an embodiment of the present application.

[0059] Fig.15 This is a structural block diagram of a foreground object extraction device in a three-dimensional scene provided by another embodiment of the present application.

[0060] Fig.16 It is a schematic diagram of the hardware structure of the electronic device provided in the embodiment of the present application. DETAILED DESCRIPTION

[0061] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0062] It should be noted that although the functional modules are divided in the device schematic and the logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the device or the order in the flowchart.

[0063] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0064] First, some nouns involved in this application are analyzed:

[0065] Artificial intelligence (AI) is a new technical science that studies and develops theories, methods, technologies and application systems for simulating, extending and expanding human intelligence. AI is a branch of computer science that attempts to understand the essence of intelligence and produce a new type of intelligent machine that can respond in a similar way to human intelligence. Research in this field includes robotics, speech recognition, image recognition, natural language processing and expert systems. AI can simulate the information process of human consciousness and thinking. AI is also a theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.

[0066] In the process of 3D scene modeling, the separation of foreground and background of the 3D model obtained by modeling is widely used. By separating the foreground and background, the object of interest can be separated from the background as the foreground, which is helpful for subsequent rendering and analysis. For example, in the application scenario of the metaverse, each user may have the need to interact with the 3D scene, such as dragging the foreground object in the digital space, such as a chair. Or in a 3D display, different numbers of graphic primitives are allocated to the foreground target and the background to achieve a better balance between the viewing experience and bandwidth usage. Or after the foreground target object is extracted separately, it can be copied, deleted, grafted (pasted to other scenes) and other editing operations are performed on it.

[0067] In the related art, when separating foreground and background, foreground targets are extracted based on point cloud data. By identifying and separating foreground targets, foreground and background are effectively distinguished from the three-dimensional model. However, when extracting foreground targets, this method requires that the foreground targets be determined before modeling. This means that every time a new foreground target is replaced, the entire modeling process needs to be repeated, including steps such as data collection, processing, and model building. This process is time-consuming, resulting in low processing efficiency for foreground and background separation.

[0068] Based on this, the embodiments of the present application provide a method, device, equipment and storage medium for extracting foreground targets in a three-dimensional scene, using the segmentation parameters for a certain perspective obtained in an interactive process to generate a corresponding foreground target mask, and then projecting the three-dimensional Gaussian point cloud onto the imaging plane of the perspective, and selecting foreground Gaussian points based on the foreground target mask to obtain the three-dimensional foreground target. This method constructs a mapping bridge between a two-dimensional scene image and a three-dimensional Gaussian point cloud, and uses a two-dimensional foreground target mask to constrain the three-dimensional Gaussian point cloud. There is no need to perform a three-dimensional reconstruction process when selecting different foreground targets, and there is no need to select the foreground target before three-dimensional reconstruction. By obtaining the segmentation parameters through an interactive process, obtaining a mask corresponding to the foreground target based on the segmentation parameters, and then using the mask to screen the three-dimensional Gaussian point cloud, the three-dimensional foreground target can be accurately separated, and the efficiency of separation and extraction of the foreground target can be improved.

[0069] The embodiments of the present application provide a method, apparatus, device and storage medium for extracting foreground objects in a three-dimensional scene, which are specifically described through the following embodiments. First, the method for extracting foreground objects in a three-dimensional scene in the embodiments of the present application is described.

[0070] The embodiments of the present application can acquire and process relevant data based on artificial intelligence technology. Among them, artificial intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results. In other words, artificial intelligence is a comprehensive technology in computer science that attempts to understand the essence of intelligence and produce a new intelligent machine that can respond in a similar way to human intelligence. Artificial intelligence is to study the design principles and implementation methods of various intelligent machines so that the machines have the functions of perception, reasoning and decision-making.

[0071] Artificial intelligence technology is a comprehensive discipline that covers a wide range of fields, including both hardware-level and software-level technologies. The basic technologies of artificial intelligence generally include sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, mechatronics and other technologies. Artificial intelligence software technology mainly includes computer vision technology, speech processing technology, natural language processing technology, and machine learning / deep learning.

[0072] The method for extracting foreground targets in a three-dimensional scene provided in an embodiment of the present application relates to the technical field of three-dimensional image processing. The method for extracting foreground targets in a three-dimensional scene provided in an embodiment of the present application can be applied to a terminal, can be applied to a server side, or can be a computer program running in a terminal or a server side. For example, a computer program can be a native program or software module in an operating system; it can be a local (Native) application (APP, Application), that is, a program that needs to be installed in an operating system to run, such as a client that supports foreground target extraction in a three-dimensional scene, or it can be a small program, that is, a program that can be run only by downloading it to a browser environment; it can also be a small program that can be embedded in any APP. In short, the above-mentioned computer program can be an application, module or plug-in in any form. Among them, the terminal communicates with the server via a network. The method for extracting foreground targets in a three-dimensional scene can be executed by a terminal or a server, or by a terminal and a server in collaboration.

[0073] In some embodiments, the terminal may be a smart phone, a tablet computer, a laptop computer, a desktop computer or a smart watch, etc. In addition, the terminal may also be an intelligent vehicle-mounted device. The intelligent vehicle-mounted device applies the foreground target extraction method in the three-dimensional scene of this embodiment to provide related services to enhance the driving experience. The server may be an independent server, or it may be a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (Content Delivery Network, CDN), and big data and artificial intelligence platforms; it may also be a service node in a blockchain system, and each service node in the blockchain system forms a peer-to-peer (P2P, Peer To Peer, P2P) network, and the P2P protocol is an application layer protocol running on the Transmission Control Protocol (Transmission Control Protocol, TCP) protocol. The terminal and the server may be connected via Bluetooth, Universal Serial Bus (Universal Serial Bus, USB) or network communication connection methods, and this embodiment is not limited here.

[0074] The present application can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, etc. The present application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present application can also be practiced in distributed computing environments, in which tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media including storage devices.

[0075] The following describes a method for extracting a foreground object in a three-dimensional scene in an embodiment of the present application.

[0076] Figure 1 is an optional flowchart of a method for extracting a foreground object in a three-dimensional scene provided in an embodiment of the present application. Figure 1 The method may include but is not limited to steps 110 to 140. It can also be understood that this embodiment Figure 1 The order of step 110 to step 140 is not specifically limited, and the order of steps can be adjusted or some steps can be reduced or added according to actual needs.

[0077] Step 110: Acquire scene images of different perspectives acquired by a plurality of image acquisition devices pre-deployed in the target scene, and perform three-dimensional reconstruction of the target scene based on the scene images to obtain a three-dimensional Gaussian point cloud.

[0078] In one embodiment, the image acquisition device may be a camera, which is pre-deployed at different locations of the target scene, and the camera is used to acquire scene images corresponding to the target scene at different viewing angles. It is understandable that the image acquisition devices are arranged around the target scene, and their number is set according to actual needs, and their number only needs to meet the accuracy of the three-dimensional Gaussian point cloud obtained by three-dimensional reconstruction based on the corresponding multiple scene images to meet the actual needs.

[0079] Reference Figure 2 , Figure 2 It is a schematic diagram of an image acquisition device for three-dimensional reconstruction provided in an embodiment of the present application. Figure 2 The target scene is indicated by the cube space in the middle, the image acquisition device is represented by a black circle, and the object in the target scene is represented by a hollow circle. The image acquisition device takes a camera as an example. Figure 234 cameras are used as the deployment position diagram, and the number can also be 36. According to the position of the camera, it can be known that in the embodiment of the present application, the image of the object in the target scene can be acquired in all directions through multiple cameras.

[0080] In one embodiment, after obtaining the scene image collected by each image acquisition device, three-dimensional reconstruction can be performed. The process of three-dimensional reconstruction in the embodiment of the present application is specifically described as follows: first, COLMAP is used to calibrate the camera system to obtain the intrinsic parameter matrix and extrinsic parameter matrix of each image acquisition device, wherein the intrinsic parameter matrix can be focal length, principal point coordinates, etc., and the extrinsic parameter matrix can be a rotation matrix and a translation vector, and the position and orientation of the image acquisition device in space are described by the intrinsic parameter matrix and the extrinsic parameter matrix. Next, feature points are extracted from each scene image to characterize the information contained in each scene image. The feature extraction algorithm here can be a scale-invariant feature transform (SIFT) or a speeded up robust features (SURF). Then, the feature points of different perspectives are matched to determine the relative position of the feature points in the three-dimensional space, and the relative position and posture between different image acquisition devices are calculated by matching the feature points. Next, the three-dimensional coordinates of each feature point are calculated using the matching feature points and the relevant parameters of the camera, and sparse reconstruction is performed to obtain a sparse three-dimensional point cloud. The sparse point cloud is then expanded into a denser point cloud using Multi-View Stereo (MVS) technology to increase the details and accuracy of the 3D model. The surface model of the 3D scene is then constructed based on the dense point cloud data using methods such as Poisson surface reconstruction and Delaunay triangulation. Finally, the texture information of the scene image is mapped to the reconstructed 3D model to obtain a preliminary 3D Gaussian point cloud. The preliminary point cloud can also be optimized, including noise removal, hole filling, smoothing, etc., to improve the quality of the 3D Gaussian point cloud.

[0081] Step 120: In response to a target selection instruction corresponding to one of the scene images as a segmented scene image, a segmentation parameter is obtained, and the segmentation parameter and the segmented scene image are input into a target segmentation model for image segmentation to obtain a foreground target mask.

[0082] In one embodiment, referring to Figure 3 , Figure 3 It is a schematic diagram of an implementation scenario provided in an embodiment of the present application. Figure 3First, multiple image acquisition devices form an image acquisition system, and then the scene images acquired by the image acquisition devices at different perspectives are sent to the GPU server for 3D reconstruction to obtain a 3D Gaussian point cloud. At the same time, the scene images obtained by each image acquisition device are transmitted to the graphical interface for view display and selection, so that users can select scene images from any perspective according to their needs. The selected scene images here contain clear and visible complete foreground targets. When the user interactively determines a scene image from a certain perspective as a segmented scene image, the target can be selected in the segmented scene image. Among them, for a medium-sized scene, a single GPU can complete the 3D reconstruction process in about 15 to 20 minutes.

[0083] It is understandable that the above process can also be performed by a computer alone without the need for user interaction. After the foreground target is determined in advance, target detection is performed on each scene image. An image containing a complete foreground target is selected as a segmented scene image, and then a region containing the foreground target is selected in the segmented scene image to complete the target selection process, which is not limited in this embodiment. The target selection process is described in detail below using user interaction as an example.

[0084] In one embodiment, referring to Figure 4 , Figure 4 The flowchart of obtaining segmentation parameters in response to a target selection instruction corresponding to one of the scene images as a segmented scene image provided by an embodiment of the present application specifically includes the following steps:

[0085] Step 410: If one of the scene images is selected as the segmented scene image, obtain a target selection instruction corresponding to the segmented scene image.

[0086] In one embodiment, a scene image may be selected as a segmented scene image by a user in a user interface, or may be determined by a computer according to a target detection result. If one of the scene images is selected as a segmented scene image, and if there is an interaction instruction for the segmented scene image, the interaction instruction is obtained as a target selection instruction corresponding to the segmented scene image.

[0087] Step 420: Obtain the selected position coordinates based on the target selection instruction, and obtain the segmentation granularity parameters corresponding to the selected position coordinates.

[0088] In one embodiment, if the user wants to select a foreground object in the segmented scene image, the user can select the area containing the foreground object by clicking, touching, etc. At this time, the selection position coordinates corresponding to the selection process can be obtained; if it is a mouse click, the selection position coordinates are the click position; if it is a touch, the selection position coordinates are the touch position. The selection position coordinates here can indicate the position of the foreground object that the user wants to segment.

[0089] In one embodiment, after obtaining the selected position coordinates, the segmentation granularity parameters corresponding to the selected position coordinates can be determined. Among them, the initial value of the segmentation granularity parameter of the embodiment of the present application is a default value, for example, it can be 1, which is used to indicate the size of the segmentation area for the foreground target in the subsequent segmentation process. It is understandable that if there is a graphic interface, a selection interface for the segmentation granularity parameter can also be provided to the user, allowing the user to determine different segmentation granularity parameters according to their own needs. If the user does not make a choice, the default value is entered.

[0090] Step 430: Selecting the position coordinates and segmentation granularity parameters as segmentation parameters.

[0091] In one embodiment, segmentation parameters are obtained according to the selected position coordinates and segmentation granularity parameters, and the segmentation parameters are used as input values ​​for a subsequent segmentation process.

[0092] In one embodiment, after the segmentation parameters are obtained, the segmentation parameters and the segmented scene image are simultaneously input into the target segmentation model for image segmentation, thereby obtaining a foreground target mask. Figure 5 , Figure 5 The flowchart of the embodiment of the present application is to input the segmentation parameters and the segmented scene image into the target segmentation model at the same time to perform image segmentation to obtain a foreground target mask, which specifically includes the following steps:

[0093] Step 510: Input the segmentation parameters and the segmented scene image into the target segmentation model to perform image segmentation to obtain an initial foreground image.

[0094] In one embodiment, the target segmentation model may be a SAM (Segment Anything Model) large model. Figure 3 , the segmentation parameters obtained through the user interaction process are used as prompt words, the segmentation parameters and the segmented scene image are input into the target segmentation model, and the image segmentation process is performed using GPU0 to obtain the initial foreground image after segmentation. At this time, the image content of the initial foreground image is related to the segmentation granularity parameters.

[0095] Step 520: If the adjustment selection instruction for the initial foreground image is responded to, an updated segmentation parameter is generated, and the updated segmentation parameter and the segmented scene image are input into the target segmentation model for image segmentation to obtain an updated initial foreground image.

[0096] In one embodiment, the initial foreground image is displayed on the image interface. If the user is not satisfied with the image range of the initial foreground image, the user can continue to adjust the range on the image interface. At this time, when adjusting, an adjustment selection instruction for the initial foreground image can be generated by clicking or touching, and the previous segmentation parameters are updated according to the adjustment selection instruction to obtain updated segmentation parameters.

[0097] In one embodiment, if an adjustment selection instruction for the initial foreground image is responded to, the process of generating updated segmentation parameters is specifically described as: obtaining the adjustment position coordinates of the adjustment selection instruction, adjusting the segmentation granularity parameters according to the positional relationship between the adjustment position coordinates and the initial foreground image, obtaining updated granularity parameters, and selecting the position coordinates and updated granularity parameters as updated segmentation parameters.

[0098] That is to say, the adjustment selection instruction will correspond to an adjustment position coordinate. At this time, it is determined whether the segmentation granularity parameter needs to be increased or decreased based on the positional relationship between the adjustment position coordinate and the initial foreground image. After adjusting it, the updated granularity parameter is obtained, and the selected position coordinate and the updated granularity parameter are used as the updated segmentation parameter. At this time, the updated segmentation parameter is used to indicate whether the segmentation area needs to be expanded or reduced based on the selected position coordinate.

[0099] In one embodiment, whether the segmentation granularity parameter needs to be increased or decreased is determined based on the positional relationship between the adjustment position coordinates and the initial foreground image. Specifically, if the adjustment position coordinates are outside the initial foreground image, it indicates that the initial foreground image needs to be increased, so the segmentation granularity parameter is increased; if the adjustment position coordinates are inside the initial foreground image, it indicates that the initial foreground image needs to be reduced, so the segmentation granularity parameter is reduced.

[0100] It is understandable that the number of times the granularity parameter is updated can be set according to actual needs, that is, it can be adjusted multiple times.

[0101] In one embodiment, referring to Figure 6 , Figure 6 It is a schematic diagram of generating updated segmentation parameters according to an adjustment selection instruction provided in an embodiment of the present application. Figure 6In the figure, it is assumed that the foreground target that the user wants to select is the entire vase. In the first interaction process, the user clicks position D1 through the target selection instruction. The selected position coordinates obtained at this time are the coordinates of position D1. After the target segmentation model performs image segmentation, the initial foreground image obtained is the dotted area where position D1 is located, recorded as area 1. After area 1 is displayed on the image interface, the user clicks at position D2, and obtains the adjusted position coordinates, i.e., the coordinates of position D2, through the adjustment selection instruction. At this time, since position D2 is outside the initial foreground image, it indicates that the initial foreground image needs to be enlarged, so the segmentation granularity parameter is increased, and the generated initial foreground image is area 1 where position D1 is located and area 2 where position D2 is located. After displaying area 1 + area 2 on the image interface, if the user still needs to expand the range, the user can click on position D3 and obtain the adjusted position coordinates, that is, the coordinates of position D3, by adjusting the selection instruction. At this time, since position D3 is outside the initial foreground image (area 1 + area 2), it indicates that the initial foreground image needs to be enlarged, so the segmentation granularity parameter is increased, and the generated initial foreground image is area 1 where position D1 is located, area 2 where position D2 is located, and area 3 where position D3 is located.

[0102] In one embodiment, if during the first interaction, the user clicks on position D1 through the target selection instruction, then after the target segmentation model performs image segmentation, the initial foreground image obtained is region 1 + region 2. After region 1 + region 2 are displayed on the image interface, if the user only wants the flower pot, he can click on position D2 and obtain the adjusted position coordinates, i.e., the coordinates of position D2, by adjusting the selection instruction. At this time, since position D2 is within the initial foreground image, it indicates that the initial foreground image needs to be reduced, so the segmentation granularity parameter is reduced, and the generated initial foreground image is region 1 where position D1 is located.

[0103] Through the above adjustment process, the user can select the desired foreground target according to actual needs.

[0104] Step 530: If a stop segmentation instruction for the initial foreground image is responded to, a foreground object mask is generated according to the initial foreground image; otherwise, segmentation parameters are continuously acquired and updated until a stop segmentation instruction is responded to.

[0105] In one embodiment, if the user is satisfied with the segmentation result of the initial foreground image, an instruction to end the segmentation can be generated, such as displaying a "confirm" symbol on the image interface. When the user clicks or touches "confirm", an instruction to stop segmentation for the initial foreground image is generated. If a stop segmentation instruction is detected, it is determined that the initial foreground image at this time meets the requirements, and a corresponding foreground target mask can be generated based on the current initial foreground image. If no stop segmentation instruction is received, it is considered that it is still in the adjustment stage, and the update of the segmentation parameters can be responded to and the corresponding segmentation operation can be performed. It is understandable that if the stop segmentation instruction is not received within the preset time, it is considered that it can be stopped when the preset time arrives, and the corresponding foreground target mask can be generated.

[0106] It is understandable that the above process can also be executed by a computer alone without user interaction, and the computer generates prediction indicators about the initial foreground image, determines whether to update the segmentation parameters and whether to stop segmentation based on the prediction indicators, and generates a foreground target mask.

[0107] In one embodiment, referring to Figure 7 , Figure 7 This is a flowchart of generating a foreground target mask according to an initial foreground image provided by an embodiment of the present application, which specifically includes the following steps:

[0108] Step 710: binarize the initial foreground image to obtain an initial foreground mask.

[0109] In one embodiment, the grayscale value of the area corresponding to the initial foreground image in the segmented scene image is set to 255, and the grayscale values ​​of other areas are set to 0, so as to obtain a binary initial foreground image, that is, an initial foreground mask.

[0110] Step 720: Obtain a preset expansion rate, and expand the initial foreground mask based on the preset expansion rate to obtain a foreground target mask.

[0111] In one embodiment, in order to offset the impact of over-segmentation on the result, a preset expansion rate is set in advance according to actual needs, and then the initial foreground mask is expanded based on the preset expansion rate. That is to say, the area is expanded along the initial foreground mask, and a circle of pixels with a gray value of 255 is added to obtain the foreground target mask.

[0112] Through the above process, a two-dimensional foreground target-related foreground target mask is obtained.

[0113] Step 130: Project the three-dimensional Gaussian point cloud onto the imaging plane corresponding to the segmented scene image to obtain the projection point corresponding to each Gaussian point in the three-dimensional Gaussian point cloud, and select some Gaussian points as foreground Gaussian points based on the projection position of the projection point and the foreground target mask.

[0114] In one embodiment, referring to Figure 8 , Figure 8 The flowchart of projecting a three-dimensional Gaussian point cloud onto an imaging plane corresponding to a segmented scene image to obtain a projection point corresponding to each Gaussian point in the three-dimensional Gaussian point cloud provided by an embodiment of the present application specifically includes the following steps:

[0115] Step 810: Obtain the intrinsic parameter matrix and extrinsic parameter matrix of the image acquisition device corresponding to the segmented scene image.

[0116] In one embodiment, the intrinsic parameter matrix K is a 3x3 matrix including parameters such as focal length and principal point coordinates. The extrinsic parameter matrix can be a 3x3 rotation matrix R and a 3x1 translation vector T. The position and orientation of the image acquisition device in space are described by the intrinsic parameter matrix and the extrinsic parameter matrix.

[0117] Step 820: Obtain the spatial coordinate value of the Gaussian point, subtract the translation vector from the spatial coordinate value to obtain a first intermediate value, calculate the product of the first intermediate value and the rotation matrix, and obtain a second intermediate value.

[0118] In one embodiment, each point in the three-dimensional Gaussian point cloud is called a Gaussian point. For each Gaussian point, the spatial coordinate value P of the Gaussian point is obtained. The homogeneous coordinates of the Gaussian point are expressed as [x, y, z, 1] T , so the first intermediate value is expressed as: PT, and the second intermediate value Pcam is expressed as: (PT)*R, where the second intermediate value is used to represent the process of converting the three-dimensional Gaussian point p from the world coordinate system to the camera coordinate system.

[0119] Step 830: Obtain the projection position of the Gaussian point on the imaging plane based on the second intermediate value, the intrinsic parameter matrix and the translation vector, and determine the projection point based on the projection position.

[0120] In one embodiment, the two-dimensional coordinate P'=[x, y] corresponding to the projection position of the projection point is expressed as:

[0121] [x,y,1]=K(Pcam.T) / Pcam[2]

[0122] The above process represents the conversion of the second intermediate value Pcam in the camera coordinate system into a two-dimensional point [x, y] on the imaging plane. Here, Pcam.T represents the transpose of Pcam, K(Pcam.T) is the process of applying the intrinsic parameter matrix, and dividing by Pcam[2] (i.e., the third component of Pcam) is to convert the three-dimensional coordinates into two-dimensional coordinates.

[0123] Through the above steps, the process uses matrix operations to project the center of the Gaussian point onto a two-dimensional imaging plane. Each Gaussian point has a corresponding projection point, and the projection points form a projection surface on the imaging plane, which contains each projection point.

[0124] In one embodiment, referring to Fig. 9 , Fig. 9 It is a schematic diagram of the projection points provided in the embodiment of the present application. Assuming that there are two image acquisition devices, whose imaging planes are imaging plane 1 and imaging plane 2, respectively, the three-dimensional Gaussian point cloud is projected to obtain the projection surface corresponding to each imaging plane.

[0125] In one embodiment, after projection, some Gaussian points can be selected as foreground Gaussian points based on the projection position of the projection point and the foreground target mask. The specific process is described as follows: the foreground area of ​​the segmented scene image is obtained according to the foreground target mask; when the projection position of the projection point is located in the foreground area, the Gaussian point corresponding to the projection point is selected as the foreground Gaussian point.

[0126] Since the foreground target mask is located on the imaging plane, the projection point is also located on the imaging plane, so mask filtering can be performed on the imaging plane. Figure 3 ,Mask filtering can be performed by GPU2, and one iteration of a single view takes about 15ms.

[0127] Among them, the foreground area of ​​the segmented scene image is obtained according to the foreground target mask. The grayscale value in the foreground area is 255, and the rest is the background area. The grayscale value of the background area is 0. If the projection position of the projection point, that is, the two-dimensional coordinate P', is located in the foreground area, then the Gaussian point is considered to be a foreground Gaussian point, otherwise it is a background Gaussian point. The whole process is expressed as:

[0128] indices=mask[p′[:,1],p′[:,0]]>0

[0129] fg = gs[indices]

[0130] Among them, indices = mask[p′[:,1],p′[:,0]]>0 means creating a Boolean index array that marks which projection positions are in the foreground area of ​​the foreground target mask. Here [p′[:,1], p′[:,0] represent the y and x coordinates of the projection point, respectively. If the value of the foreground target mask at these coordinate positions is greater than 0, the corresponding index is true (that is, it belongs to the foreground). And fg = gs[indices] uses the above Boolean index array to select the foreground Gaussian points belonging to the foreground from the three-dimensional Gaussian point cloud gs. Here, gs[indices] means selecting those elements in gs corresponding to indices being true.

[0131] Through the above process, a set of foreground Gaussian points corresponding to the foreground target is screened from the three-dimensional Gaussian point cloud.

[0132] In one embodiment, referring to Fig.10 , Fig.10 Schematic diagram of the mask filtering process of foreground Gaussian points provided in the embodiment of the present application. Assuming that the three-dimensional Gaussian point cloud is a cube, at this time, the scene image corresponding to the image acquisition device T1 is a segmented scene image, and the obtained foreground target mask is C1 in the figure. After mask filtering, the three-dimensional Gaussian point cloud is segmented into point cloud set Q1, point cloud set Q2 and point cloud set Q3, and point cloud set Q2 is used as the set of foreground Gaussian points.

[0133] Step 140: Obtain a three-dimensional foreground object based on the foreground Gaussian points.

[0134] In one embodiment, referring to Fig.11 , Fig.11 This is a flow chart of obtaining a three-dimensional foreground target based on foreground Gaussian points provided in an embodiment of the present application, which specifically includes the following steps:

[0135] Step 1110: perform at least one iteration process. In each iteration, select scene images corresponding to other perspectives as segmented scene images, use the foreground Gaussian points of the previous iteration as the three-dimensional Gaussian point cloud, and update the foreground Gaussian points.

[0136] In one embodiment, since the set of foreground Gaussian points obtained in a single iteration may contain point clouds of some non-foreground targets, it needs to be further segmented. At this time, another iterative process is performed, and scene images corresponding to other perspectives different from the previously segmented scene images are selected as the segmented scene images again. The corresponding set of foreground Gaussian points of the previous iteration is used as the new three-dimensional Gaussian point cloud, and the above-mentioned segmentation process is performed to obtain updated foreground Gaussian points.

[0137] In one embodiment, referring to Fig.12 , Fig.12 A schematic diagram of the iterative process of updating the foreground Gaussian points provided in the embodiment of the present application. Fig.10 Based on the point cloud set Q2 corresponding to the foreground Gaussian points obtained by segmentation, the point cloud set Q2 is used as a new three-dimensional Gaussian point cloud, and the scene image corresponding to the new perspective T2 is selected as the segmented scene image. The obtained foreground target mask is C2 in the figure. At this time, segmentation is performed on the point cloud set Q2, and the three-dimensional Gaussian point cloud is segmented into point cloud set Q21, point cloud set Q22 and point cloud set Q23. The point cloud set Q22 is used as a set of updated foreground Gaussian points.

[0138] It is understandable that, referring to Figure 3 After rendering the point cloud obtained in each iteration, the rendering result can be displayed on the image interface to allow the user to determine whether to continue the iteration. Alternatively, the computer can determine whether to continue the iteration based on the point cloud result.

[0139] Step 1120: Obtain a three-dimensional foreground object based on the foreground Gaussian points obtained in the last iteration.

[0140] In one embodiment, the number of iterations can be set according to actual needs, or can be terminated by a user interaction process in the image interface, and the foreground Gaussian points obtained in the last iteration are used as the three-dimensional foreground targets. Since each iteration performs point cloud filtering on the basis of the previous iteration, it can also be understood that the three-dimensional foreground target obtained in the end is the intersection of the results of each iteration.

[0141] In one embodiment, referring to Figure 3 , a 3D foreground target is obtained through the above process, and a 3D Gaussian point cloud other than the 3D foreground target is used as a 3D background point cloud. The 3D foreground target and the 3D background point cloud can be saved and rendered separately. The 3D foreground target can be displayed on the image interface after being rendered separately, and further scene editing operations such as copying, deleting, moving, etc. can be performed through user interaction on the image interface, which is not limited in this embodiment.

[0142] In one embodiment, referring to Fig.13 , Fig.13 It is an overall flow chart of the method for extracting foreground objects in a three-dimensional scene provided in an embodiment of the present application.

[0143] First, multi-view image input is performed, and scene images of different viewpoints acquired by multiple image acquisition devices pre-deployed in the target scene are acquired, and the target scene is reconstructed in three dimensions based on the scene images. During the three-dimensional reconstruction process, the sparse point cloud and camera parameters are calculated to obtain a three-dimensional Gaussian point cloud. At the same time, the scene image obtained by each image acquisition device is transmitted to the graphical interface for view display and selection, so that users can select the scene image of any viewpoint according to their needs.

[0144] Next, when the user determines a scene image of a certain perspective as a segmented scene image through interaction, the target selection can be performed in the segmented scene image. Perform foreground target segmentation in a two-dimensional view, obtain the selected position coordinates based on the target selection instruction corresponding to the segmented scene image, and obtain the segmentation granularity parameter corresponding to the selected position coordinates, use the selected position coordinates and the segmentation granularity parameter as segmentation parameters, and input the segmentation parameters and the segmented scene image into the target segmentation model for image segmentation to obtain the initial foreground image. In this process, if there is user interaction, the adjusted position coordinates of the adjustment selection instruction in the interaction process are obtained. If the adjusted position coordinates are outside the initial foreground image, the segmentation granularity parameter is increased; if the adjusted position coordinates are within the initial foreground image, the segmentation granularity parameter is reduced. The selected position coordinates and the updated granularity parameter are used as updated segmentation parameters, and the updated segmentation parameters and the segmented scene image are input into the target segmentation model for image segmentation to obtain an updated initial foreground image. If the stop segmentation instruction for the initial foreground image is responded to, a foreground target mask is generated according to the initial foreground image, otherwise the updated segmentation parameters are continuously obtained until the stop segmentation instruction is responded to.

[0145] Then the 3D Gaussian point cloud is projected onto the imaging plane corresponding to the segmented scene image to obtain the projection point corresponding to each Gaussian point in the 3D Gaussian point cloud. Based on the projection position of the projection point and the foreground target mask, it is determined whether the projection point is within the mask. Based on the judgment result, some Gaussian points are selected as foreground Gaussian points, and the rest are selected as background Gaussian points. Finally, the 3D foreground target is obtained based on the foreground Gaussian points, and the 3D background point cloud is obtained based on the background Gaussian points.

[0146] The method for extracting foreground targets in a three-dimensional scene in an embodiment of the present application utilizes the central projection method of Gaussian points to construct a mapping bridge between a two-dimensional scene image and a three-dimensional Gaussian point cloud, so that the prior knowledge obtained from the simple and easy-to-obtain two-dimensional scene image can constrain the three-dimensional Gaussian point cloud. At the same time, the embodiment of the present application utilizes the geometric consistency between multiple image acquisition devices so that the prior information of scene images from different perspectives can act on the same set of three-dimensional Gaussian point clouds. In theory, the more perspectives involved in the iteration, the better the final extraction effect of the three-dimensional foreground target. In addition, the embodiment of the present application utilizes an image interface to introduce a user interaction process to select views and foreground targets. The stability of this method is much higher than that of a non-interactive foreground extraction solution, and supports the extraction of any category of three-dimensional foreground targets.

[0147] In one embodiment, referring to Fig.14 , Fig.14 It is a schematic diagram of a foreground object extraction method in a three-dimensional scene provided in an embodiment of the present application.

[0148] First, the three-dimensional scene point cloud includes multiple circles of tables, flowers, and backgrounds. At this time, it is assumed that the extracted foreground target is a flower (including a vase). After the first iteration process, a perspective is selected to obtain a foreground target mask. The foreground Gaussian points obtained after point cloud segmentation also include some point clouds related to the table and background. Therefore, the second iteration process is performed, and the scene image corresponding to other perspectives is selected as the segmented scene image to generate the foreground target mask of the second iteration. The set of foreground Gaussian points of the first iteration is used as the three-dimensional Gaussian point cloud, and the point cloud segmentation is performed. The foreground Gaussian points are updated. At this time, it is found that the set of foreground Gaussian points obtained basically does not contain irrelevant point clouds. The set of foreground Gaussian points at this time is used as the three-dimensional foreground target.

[0149] The technical solution provided by the embodiment of the present application is to obtain scene images of different perspectives obtained by multiple image acquisition devices pre-deployed in the target scene, and to perform three-dimensional reconstruction of the target scene based on the scene images to obtain a three-dimensional Gaussian point cloud; to obtain segmentation parameters in response to a target selection instruction corresponding to one of the scene images as a segmented scene image, and to input the segmentation parameters and the segmented scene image into the target segmentation model for image segmentation to obtain a foreground target mask; to project the three-dimensional Gaussian point cloud onto the imaging plane corresponding to the segmented scene image to obtain the projection points corresponding to each Gaussian point in the three-dimensional Gaussian point cloud, and to select some Gaussian points as foreground Gaussian points based on the projection positions of the projection points and the foreground target mask; and to obtain a three-dimensional foreground target based on the foreground Gaussian points. The embodiment of the present application generates a corresponding foreground target mask using the segmentation parameters for a certain perspective obtained through the interactive process, and then projects the three-dimensional Gaussian point cloud onto the imaging plane of the perspective, and selects foreground Gaussian points based on the foreground target mask to obtain a three-dimensional foreground target. This method builds a mapping bridge between the two-dimensional scene image and the three-dimensional Gaussian point cloud, and uses the two-dimensional foreground target mask to constrain the three-dimensional Gaussian point cloud. There is no need to perform the three-dimensional reconstruction process when selecting different foreground targets, nor is there a need to select the foreground target before three-dimensional reconstruction. The segmentation parameters are obtained through the interactive process, and the mask corresponding to the foreground target is obtained based on the segmentation parameters. The mask is then used to screen the three-dimensional Gaussian point cloud, which can accurately separate the three-dimensional foreground target and improve the separation and extraction efficiency of the foreground target.

[0150] The embodiment of the present application also provides a foreground object extraction device in a three-dimensional scene, which can implement the foreground object extraction method in the three-dimensional scene. Fig.15 , the device comprises:

[0151] The three-dimensional reconstruction module 1510 is used to obtain scene images of different perspectives acquired by multiple image acquisition devices pre-deployed in the target scene, and to perform three-dimensional reconstruction of the target scene based on the scene images to obtain a three-dimensional Gaussian point cloud.

[0152] The two-dimensional segmentation module 1520 is used to obtain segmentation parameters in response to a target selection instruction corresponding to one of the scene images as a segmented scene image, input the segmentation parameters and the segmented scene image into a target segmentation model for image segmentation, and obtain a foreground target mask.

[0153] Mask filtering module 1530: used to project the three-dimensional Gaussian point cloud onto the imaging plane corresponding to the segmented scene image, obtain the projection point corresponding to each Gaussian point in the three-dimensional Gaussian point cloud, and select some Gaussian points as foreground Gaussian points based on the projection position of the projection point and the foreground target mask.

[0154] The target generation module 1540 is used to obtain a three-dimensional foreground target based on the foreground Gaussian points.

[0155] The specific implementation of the foreground object extraction device in a three-dimensional scene of this embodiment is basically the same as the specific implementation of the foreground object extraction method in a three-dimensional scene described above, and will not be repeated here.

[0156] The present application also provides an electronic device, including:

[0157] at least one memory; at least one processor; at least one program;

[0158] The program is stored in the memory, and the processor executes the at least one program to implement the foreground object extraction method in the three-dimensional scene implemented in the present application. The electronic device can be any intelligent terminal including a mobile phone, a tablet computer, a personal digital assistant (PDA), a car computer, etc.

[0159] See also Fig.16 , Fig.16 The hardware structure of an electronic device of another embodiment is illustrated, and the electronic device includes:

[0160] The processor 1601 may be implemented by a general-purpose central processing unit (CPU), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present application;

[0161] The memory 1602 can be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 1602 can store an operating system and other application programs. When the technical solution provided in the embodiment of this specification is implemented by software or firmware, the relevant program code is stored in the memory 1602, and the processor 1601 calls and executes the foreground object extraction method in a three-dimensional scene of the embodiment of this application;

[0162] Input / output interface 1603, used to implement information input and output;

[0163] Communication interface 1604, used to realize communication interaction between the device and other devices, which can be realized by wired means (such as USB, network cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.); and

[0164] A bus 1605 that transmits information between various components of the device (e.g., the processor 1601, the memory 1602, the input / output interface 1603, and the communication interface 1604);

[0165] The processor 1601 , the memory 1602 , the input / output interface 1603 and the communication interface 1604 are connected to each other in communication within the device via the bus 1605 .

[0166] An embodiment of the present application further provides a storage medium, which is a storage medium storing a computer program. When the computer program is executed by a processor, the foreground target extraction method in the three-dimensional scene is implemented.

[0167] As a non-transitory storage medium, the memory can be used to store non-transitory software programs and non-transitory computer executable programs. In addition, the memory may include a high-speed random access memory, and may also include a non-transitory memory, such as at least one disk storage device, a flash memory device, or other non-transitory solid-state storage device. In some embodiments, the memory may optionally include a memory remotely arranged relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0168] The foreground target extraction method, device, equipment and storage medium in a three-dimensional scene proposed in the embodiment of the present application obtains scene images of different perspectives obtained by multiple image acquisition devices pre-deployed in the target scene, and performs three-dimensional reconstruction of the target scene based on the scene images to obtain a three-dimensional Gaussian point cloud; obtains segmentation parameters in response to a target selection instruction corresponding to one of the scene images as a segmented scene image, inputs the segmentation parameters and the segmented scene image into the target segmentation model for image segmentation, and obtains a foreground target mask; projects the three-dimensional Gaussian point cloud onto the imaging plane corresponding to the segmented scene image to obtain the projection points corresponding to each Gaussian point in the three-dimensional Gaussian point cloud, selects some Gaussian points as foreground Gaussian points based on the projection positions of the projection points and the foreground target mask; and obtains a three-dimensional foreground target based on the foreground Gaussian points. The embodiment of the present application generates a corresponding foreground target mask using the segmentation parameters for a certain perspective obtained in the interactive process, then projects the three-dimensional Gaussian point cloud onto the imaging plane of the perspective, and selects foreground Gaussian points based on the foreground target mask to obtain a three-dimensional foreground target. This method builds a mapping bridge between the two-dimensional scene image and the three-dimensional Gaussian point cloud, and uses the two-dimensional foreground target mask to constrain the three-dimensional Gaussian point cloud. There is no need to perform the three-dimensional reconstruction process when selecting different foreground targets, nor is there a need to select the foreground target before three-dimensional reconstruction. The segmentation parameters are obtained through the interactive process, and the mask corresponding to the foreground target is obtained based on the segmentation parameters. The mask is then used to screen the three-dimensional Gaussian point cloud, which can accurately separate the three-dimensional foreground target and improve the separation and extraction efficiency of the foreground target.

[0169] The embodiments described in the embodiments of the present application are intended to more clearly illustrate the technical solutions of the embodiments of the present application and do not constitute a limitation on the technical solutions provided in the embodiments of the present application. Those skilled in the art will appreciate that with the evolution of technology and the emergence of new application scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0170] Those skilled in the art will appreciate that the technical solutions shown in the figures do not constitute a limitation on the embodiments of the present application, and may include more or fewer steps than shown in the figures, or a combination of certain steps, or different steps.

[0171] The device embodiments described above are merely illustrative, and the units described as separate components may or may not be physically separated, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0172] Those skilled in the art will appreciate that all or some of the steps in the methods disclosed above, and the functional modules / units in the systems and devices may be implemented as software, firmware, hardware, or a suitable combination thereof.

[0173] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0174] It should be understood that in the present application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0175] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the above units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0176] The units described above as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0177] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0178] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including multiple instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), disk or optical disk and other media that can store programs.

[0179] The preferred embodiments of the present invention are described above with reference to the accompanying drawings, but the scope of the rights of the present invention is not limited thereto. Any modification, equivalent substitution and improvement made by a person skilled in the art without departing from the scope and essence of the present invention should be within the scope of the rights of the present invention.

Claims

1. A method for extracting foreground objects in a three-dimensional scene, characterized in that: include: Acquire scene images of different perspectives acquired by a plurality of image acquisition devices pre-deployed in the target scene, and perform three-dimensional reconstruction of the target scene based on the scene images to obtain a three-dimensional Gaussian point cloud; In response to a target selection instruction corresponding to one of the scene images as a segmented scene image, a segmentation parameter is obtained, and the segmentation parameter and the segmented scene image are input into a target segmentation model for image segmentation to obtain a foreground target mask; Projecting the three-dimensional Gaussian point cloud onto an imaging plane corresponding to the segmented scene image to obtain projection points corresponding to each Gaussian point in the three-dimensional Gaussian point cloud, and selecting part of the Gaussian points as foreground Gaussian points based on the projection positions of the projection points and the foreground target mask; A three-dimensional foreground object is obtained based on the foreground Gaussian points.

2. The method for extracting foreground objects in a three-dimensional scene according to claim 1, characterized in that: The obtaining of a three-dimensional foreground object based on the foreground Gaussian point comprises: Perform at least one iteration process, in each iteration, select the scene image corresponding to other perspectives as the segmented scene image, use the foreground Gaussian point of the previous iteration as the three-dimensional Gaussian point cloud, and update the foreground Gaussian point; A three-dimensional foreground object is obtained based on the foreground Gaussian points obtained in the last iteration.

3. The method for extracting foreground objects in a three-dimensional scene according to claim 1, characterized in that: The obtaining of the segmentation parameters in response to the target selection instruction corresponding to one of the scene images as the segmented scene image comprises: If one of the scene images is selected as the segmented scene image, obtaining a target selection instruction corresponding to the segmented scene image; Obtaining the selected position coordinates based on the target selection instruction, and obtaining the segmentation granularity parameters corresponding to the selected position coordinates; The selected position coordinates and the segmentation granularity parameters are used as the segmentation parameters.

4. The method for extracting foreground objects in a three-dimensional scene according to claim 3, characterized in that: The step of inputting the segmentation parameters and the segmented scene image into a target segmentation model to perform image segmentation to obtain a foreground target mask comprises: Inputting the segmentation parameters and the segmented scene image into a target segmentation model to perform image segmentation to obtain an initial foreground image; If an adjustment selection instruction for the initial foreground image is responded to, an updated segmentation parameter is generated, and the updated segmentation parameter and the segmented scene image are input into the target segmentation model for image segmentation to obtain an updated initial foreground image; If a stop segmentation instruction for the initial foreground image is responded to, the foreground object mask is generated according to the initial foreground image; otherwise, the updated segmentation parameters are continuously acquired until a stop segmentation instruction is responded to.

5. The method for extracting foreground objects in a three-dimensional scene according to claim 4, characterized in that: If the adjustment selection instruction for the initial foreground image is responded to, the updated segmentation parameters are generated, including: Acquire the adjustment position coordinates of the adjustment selection instruction, and adjust the segmentation granularity parameter according to the positional relationship between the adjustment position coordinates and the initial foreground image to obtain an updated granularity parameter; The selected position coordinates and the update granularity parameter are used as the update segmentation parameter.

6. The method for extracting foreground objects in a three-dimensional scene according to claim 5, characterized in that: The step of adjusting the segmentation granularity parameter according to the positional relationship between the adjusted position coordinates and the initial foreground image to obtain an updated granularity parameter includes: If the adjusted position coordinates are outside the initial foreground image, increasing the segmentation granularity parameter; If the adjusted position coordinates are within the initial foreground image, the segmentation granularity parameter is reduced.

7. The method for extracting foreground objects in a three-dimensional scene according to claim 4, characterized in that: The step of generating the foreground object mask according to the initial foreground image comprises: Binarizing the initial foreground image to obtain an initial foreground mask; A preset expansion rate is obtained, and the initial foreground mask is expanded based on the preset expansion rate to obtain the foreground object mask.

8. The method for extracting foreground objects in a three-dimensional scene according to any one of claims 1 to 7, characterized in that: The step of projecting the three-dimensional Gaussian point cloud onto an imaging plane corresponding to the segmented scene image to obtain a projection point corresponding to each Gaussian point in the three-dimensional Gaussian point cloud includes: Acquire an intrinsic parameter matrix and an extrinsic parameter matrix of an image acquisition device corresponding to the segmented scene image, wherein the extrinsic parameter matrix includes a rotation matrix and a translation vector; Acquire the spatial coordinate value of the Gaussian point, subtract the translation vector from the spatial coordinate value to obtain a first intermediate value, and calculate the product of the first intermediate value and the rotation matrix to obtain a second intermediate value; The projection position of the Gaussian point on the imaging plane is obtained based on the second intermediate value, the intrinsic parameter matrix and the translation vector, and the projection point is determined based on the projection position.

9. The method for extracting foreground objects in a three-dimensional scene according to claim 8, characterized in that: The selecting part of the Gaussian points as foreground Gaussian points based on the projection positions of the projection points and the foreground object mask comprises: Obtaining a foreground area of ​​the segmented scene image according to the foreground object mask; When the projection position of the projection point is located in the foreground area, a Gaussian point corresponding to the projection point is selected as the foreground Gaussian point.

10. A device for extracting foreground objects in a three-dimensional scene, characterized in that: include: A three-dimensional reconstruction module is used to obtain scene images of different perspectives acquired by multiple image acquisition devices pre-deployed in the target scene, and to perform three-dimensional reconstruction of the target scene based on the scene images to obtain a three-dimensional Gaussian point cloud; A two-dimensional segmentation module: used for obtaining segmentation parameters in response to a target selection instruction corresponding to one of the scene images as a segmented scene image, inputting the segmentation parameters and the segmented scene image into a target segmentation model for image segmentation, and obtaining a foreground target mask; A mask filtering module is used to project the three-dimensional Gaussian point cloud onto an imaging plane corresponding to the segmented scene image to obtain projection points corresponding to each Gaussian point in the three-dimensional Gaussian point cloud, and select part of the Gaussian points as foreground Gaussian points based on the projection positions of the projection points and the foreground target mask; Target generation module: used to obtain a three-dimensional foreground target based on the foreground Gaussian points.

11. An electronic device, characterized in that: The electronic device comprises a memory and a processor, wherein the memory stores a computer program, and the processor implements the method for extracting a foreground object in a three-dimensional scene as claimed in any one of claims 1 to 9 when executing the computer program.

12. A storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the method for extracting a foreground object in a three-dimensional scene according to any one of claims 1 to 9 is implemented.

Images