Image processing method, model training method and related equipment

Abstract

The present invention provides an image processing method, a model training method and related equipment, which relate to the field of computer vision. The image processing method includes: performing target processing on an image set to generate a three-dimensional Gaussian radiation field, wherein the image set includes images corresponding to a first object at different viewing angles, and the target processing is used to associate feature labels in images corresponding to the first object at different viewing angles; based on the target grid scale, the three-dimensional Gaussian radiation field is spatially divided to obtain multiple grid spaces; based on the feature vectors of the grid vertices of each grid space, the three-dimensional Gaussian radiation field is feature segmented. Compared with traditional segmentation methods, the amount of data to be processed is reduced, and the efficiency of three-dimensional scene segmentation is improved.

Description

Technical Field

[0001] The present invention relates to the field of computer vision, and in particular to an image processing method, a model training method and related equipment. Background Art

[0002] 3D scene segmentation is an important task in the field of computer vision. It involves dividing objects in 3D space into groups, each of which corresponds to an object in the real world. The main applications of 3D scene segmentation can include autonomous driving, robot navigation, environmental monitoring, urban planning, and augmented reality and virtual reality.

[0003] However, existing 3D scene segmentation is usually based on segmentation methods of traditional deep learning frameworks. The data used by such methods generally comes from point clouds collected by lidar. The amount of data generated by lidar is usually very large and requires high-performance computing resources for processing and storage.

[0004] It can be seen that the existing three-dimensional scene segmentation process has the problem of large data processing volume. Summary of the invention

[0005] The embodiments of the present invention provide an image processing method, a model training method and related equipment to solve the problem of large amount of data processing in the existing three-dimensional scene segmentation process.

[0006] To solve the above technical problems, the present invention is achieved as follows:

[0007] In a first aspect, an embodiment of the present invention provides an image processing method, the method comprising:

[0008] Performing target processing on an image set to generate a three-dimensional Gaussian radiation field, the image set including images corresponding to a first object at different viewing angles, the target processing being used to associate feature labels in the images corresponding to the first object at different viewing angles;

[0009] Based on the target grid scale, the three-dimensional Gaussian radiation field is spatially divided to obtain a plurality of grid spaces;

[0010] Based on the feature vectors of the grid vertices of each grid space, feature segmentation is performed on the three-dimensional Gaussian radiation field.

[0011] In a second aspect, an embodiment of the present invention provides a model training method, the method comprising:

[0012] The first module and the second module of the initial model are iteratively trained based on a training image set, wherein the training image set includes images corresponding to multiple objects at different viewing angles, and the pixel positions in the images corresponding to the same object among the multiple objects at different viewing angles are marked by two-dimensional segmentation labels, the first module is used to generate a three-dimensional Gaussian radiation field, and the second module is used to perform feature segmentation on the three-dimensional Gaussian radiation field.

[0013] In a third aspect, an embodiment of the present invention provides an image processing device, the device comprising:

[0014] a processing module, configured to perform target processing on an image set to generate a three-dimensional Gaussian radiation field, wherein the image set includes images corresponding to a first object at different viewing angles, and the target processing is configured to associate feature labels in images corresponding to the first object at different viewing angles;

[0015] A partitioning module, used for spatially partitioning the three-dimensional Gaussian radiation field based on a target grid scale to obtain a plurality of grid spaces;

[0016] The segmentation module is used to perform feature segmentation on the three-dimensional Gaussian radiation field based on the feature vectors of the grid vertices in each grid space.

[0017] In a fourth aspect, an embodiment of the present invention provides a model training device, the device comprising:

[0018] A training module is used to iteratively train the first module and the second module of the initial model based on a training image set, wherein the training image set includes images corresponding to multiple objects at different viewing angles, and the pixel positions in the images corresponding to the same object among the multiple objects at different viewing angles are marked by two-dimensional segmentation labels, the first module is used to generate a three-dimensional Gaussian radiation field, and the second module is used to perform feature segmentation on the three-dimensional Gaussian radiation field.

[0019] In a fifth aspect, an embodiment of the present invention provides an electronic device, including:

[0020] at least one processor; and

[0021] a memory communicatively connected to the at least one processor; wherein,

[0022] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the steps of the method as described in the first aspect, or to enable the at least one processor to perform the steps of the method as described in the second aspect.

[0023] In a sixth aspect, an embodiment of the present invention provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the method described in the first aspect are implemented, or the at least one processor is enabled to execute the steps of the method described in the second aspect.

[0024] In a seventh aspect, an embodiment of the present invention provides a computer program product, comprising computer instructions, which, when executed by a processor, implement the steps of the method described in the first aspect, so that the at least one processor can execute the steps of the method described in the second aspect.

[0025] In an embodiment of the present invention, the feature labels in the images corresponding to each object in the image set at different viewing angles are first associated to improve the high fidelity of the generated three-dimensional Gaussian radiation field at any viewing angle; then the three-dimensional Gaussian radiation field is spatially divided based on the target grid scale to improve the uniformity of multiple grid spaces. The grid vertices of adjacent grid spaces in the multiple grid spaces are shared with each other, so that the shared grid vertices in adjacent grid spaces can use the same feature vector. In this way, in the process of feature segmentation of the three-dimensional Gaussian radiation field, different objects can be segmented based on the feature vectors of the grid vertices in each grid space, which reduces the amount of data to be processed compared to traditional segmentation methods and improves the efficiency of three-dimensional scene segmentation. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0027] Figure 1 is a flow chart of an image processing method provided by an embodiment of the present invention;

[0028] Figure 2 is a flow chart of a model training method provided by an embodiment of the present invention;

[0029] Figure 3 is a structural schematic diagram of an image processing device provided by an embodiment of the present invention;

[0030] Figure 4 It is a structural schematic diagram of a model training device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0031] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0032] See also Figure 1 , Figure 1 is a flow chart of an image processing method provided by an embodiment of the present invention, such as Figure 1 As shown, the method comprises the following steps:

[0033] Step 101: performing target processing on an image set to generate a three-dimensional Gaussian radiation field, wherein the image set includes images corresponding to a first object at different viewing angles, and the target processing is used to associate feature labels in the images corresponding to the first object at different viewing angles;

[0034] In this step, the image set may be a series of image data collected by a camera in a real space (such as a bedroom, a football field, a park, etc.). The same object may have different perspectives in different images in the image set. For example, the image of the goal taken from the audience seats of the football field is compared with the image of the goal taken from the penalty line of the football field. Although the goal is the same goal, the perspectives presented in these two images are different. In this example, the feature labels in the images corresponding to the first object at different perspectives are associated so that the first object has the same feature labels under multiple views, which facilitates the identification of the first object in different images. In this way, the point cloud data of the first object can be generated according to the images corresponding to the first object at different perspectives. The first object may be any one of the multiple objects included in the image set. According to the images corresponding to each object at different perspectives, its corresponding point cloud data is generated, thereby forming a three-dimensional Gaussian radiation field.

[0035] The first object may be an object in the real space or an area in the real space.

[0036] Step 102: based on the target grid scale, spatially divide the three-dimensional Gaussian radiation field to obtain a plurality of grid spaces;

[0037] The three-dimensional Gaussian has properties such as mean, rotation, scale, transparency, and color (expressed using spherical harmonic functions). In this step, the three-dimensional Gaussian radiation field can be spatially divided to obtain multiple grid spaces, and the three-dimensional Gaussian radiation field can be further feature segmented through step 103, thereby realizing three-dimensional scene segmentation. Among them, considering that the three-dimensional Gaussian radiation field is feature segmented based on the vertices of the grid space in the subsequent step 103, the multiple grid spaces have the same scale. In other words, the scale of each grid space can be the target grid scale. In this way, the three-dimensional Gaussian radiation field is spatially divided based on the target grid scale, which improves the uniformity of multiple grid spaces. The grid vertices of adjacent grid spaces in multiple grid spaces are shared with each other, so that the shared grid vertices in adjacent grid spaces can use the same feature vector, thereby reducing the amount of data to be processed.

[0038] Step 103: perform feature segmentation on the three-dimensional Gaussian radiation field based on the feature vectors of the grid vertices in each grid space.

[0039] In this step, each grid space corresponds to 8 grid vertices, and each grid vertex is assigned a feature vector of multiple dimensions (for example, 16 dimensions); in the multiple grid spaces obtained after spatial division of the three-dimensional Gaussian radiation field, each grid space contains a certain number of three-dimensional Gaussians. According to the feature vectors of the 8 grid vertices in a grid space, the feature vector of any three-dimensional Gaussian in the grid space can be determined, and then the three-dimensional Gaussian radiation field can be feature segmented according to the feature vectors of each three-dimensional Gaussian. In this way, in the process of determining each three-dimensional Gaussian feature vector, the feature vector of the shared grid vertex can be used for calculation, which reduces resource usage and improves calculation efficiency.

[0040] In an embodiment of the present invention, the feature labels in the images corresponding to each object in the image set at different viewing angles are first associated to improve the high fidelity of the generated three-dimensional Gaussian radiation field at any viewing angle; then the three-dimensional Gaussian radiation field is spatially divided based on the target grid scale to improve the uniformity of multiple grid spaces. The grid vertices of adjacent grid spaces in the multiple grid spaces are shared with each other, so that the shared grid vertices in adjacent grid spaces can use the same feature vector. In this way, in the process of feature segmentation of the three-dimensional Gaussian radiation field, different objects can be segmented based on the feature vectors of the grid vertices in each grid space, which reduces the amount of data to be processed compared to traditional segmentation methods and improves the efficiency of three-dimensional scene segmentation.

[0041] In some embodiments, the step 101 of performing target processing on the image set to generate a three-dimensional Gaussian radiation field includes:

[0042] Performing feature recognition processing on the image set to obtain feature labels in images corresponding to multiple objects in the image set at different viewing angles, wherein the first object is any one of the multiple objects;

[0043] Associating feature labels whose matching degree between the feature labels is greater than or equal to a preset matching threshold, to obtain two-dimensional segmentation labels corresponding to the multiple objects respectively, wherein the two-dimensional segmentation labels are used to indicate pixel positions in images corresponding to the same object among the multiple objects at different viewing angles;

[0044] The three-dimensional Gaussian radiation fields corresponding to the multiple objects are generated according to the two-dimensional segmentation labels respectively corresponding to the multiple objects.

[0045] In this embodiment, a group of multi-view images can be collected by a camera in a real space to obtain an image set. The image set can include X view images, where X is a positive integer, such as image0.jpg, image1.jpg...imageX.jpg, and these X view images can be stored in the "images" folder.

[0046] Performing feature recognition processing on each view image in the image set can be to determine the categories of different objects according to different grayscale values ​​in the view image, and perform feature marking according to different categories to obtain feature labels in the images corresponding to multiple objects in the image set at different viewing angles. Then, feature labels whose matching degree between each feature label is greater than or equal to a preset matching threshold are associated to obtain two-dimensional segmentation labels corresponding to multiple objects. The same object in different viewing angles (i.e., different view images in the image set) can be associated through the two-dimensional segmentation label. In this way, based on the correlation of the two-dimensional segmentation label, the feature information in the view image corresponding to each object at different viewing angles is comprehensively considered, thereby improving the accuracy of the generated three-dimensional Gaussian radiation field.

[0047] Before performing feature recognition processing on each view image in the image set, each view image may be subjected to preprocessing such as denoising and brightness correction.

[0048] Specifically, the image set can be processed based on the pre-trained first network model to generate a three-dimensional Gaussian radiation field. The training process of the first network model can be described as follows:

[0049] First, the training image set is preprocessed so that the training image set includes view images corresponding to multiple objects at different perspectives, and the pixel positions in the view images corresponding to the same object at different perspectives are marked by two-dimensional segmentation labels. In addition, the two-dimensional segmentation labels are optimized using the morphological erosion and dilation operation in image processing to eliminate errors and noise in the segmentation labels. All view images are named according to the order rule, such as image0.jpg, image1.jpg...imageX.jpg, and stored in the "images" folder, where X is a positive integer; the classic Structure from Motion (SFM) 3D reconstruction solution Colmap is called to perform sparse reconstruction of multi-view images. In the "feature matching process" of sparse reconstruction, the "sequential matching" strategy is used to obtain the intrinsic and extrinsic parameters of the camera.

[0050] Then, you can use the Segment anything model (SAM) to select the "automatic mode" to segment all image targets for the multi-view images stored in the "images" folder, obtain the segmented two-dimensional label image, and save it as "mask%04d.jpg", for example: mask0023.jpg. Different grayscale values ​​in the segmented image correspond to different object categories. Traverse each segmentation category and use the corresponding label value as the segmentation value to globally binarize the label image. Subsequently, the binary image is processed using the erosion and dilation operation of image morphology processing in OpenCV, with a window parameter size of 5. Finally, each binary image is recombined according to the original feature label to obtain a corrected two-dimensional label grayscale image for segmentation, which is saved as "re_mask%04d.jpg", for example: re_mask0023.jpg.

[0051] Among them, SAM performs segmentation processing on a single view image and does not consider the correlation under multiple views. The feature labels obtained are not consistent under multiple views, that is, the feature labels of the same object in different views are different. It is necessary to align the consistency of the feature labels of multiple views to obtain the two-dimensional segmentation labels corresponding to multiple objects. When Colmap estimates the camera parameters, it is necessary to perform feature matching based on the image features extracted by the feature operator. The extracted features can be reused to associate the feature labels consistently. The specific implementation steps are as follows:

[0052] Save the multi-view feature matching information in Colmap, that is, the corresponding pixel positions of the same object in multiple views. Randomly select a feature label, and start associating it based on the pixel positions in the image corresponding to the label that match the images of other views. Specifically, change the segmentation label value of the feature labeled as the same object in other multi-views to the label value of the currently selected segmented object, and mark the image as "used" for the consistent alignment of the object. Traverse all the segmentation label values ​​of the target object and all other views that are not marked as "used", and perform consistent association of feature labels as described above to obtain the two-dimensional segmentation labels corresponding to multiple objects. Finally, the feature labels of all objects are consistently associated under multiple views, that is, the multi-view images of the same object. Figure 2 The dimension split labels are the same.

[0053] Furthermore, the training method of the 3D Gaussian radiation field can be to project the 3D Gaussian in the space onto the image plane, and use alpha blending (α-blending) to rasterize all the projected 2D Gaussians under the pixel in the order from near to far in the corresponding space, and finally obtain the rendered image under a specific viewing angle. The properties of the 3D Gaussian are based on the red, green, blue (RGB) loss L1 and the structural similarity index loss (Structural similarity index, D-SSIM) L of the rendered image and the true value image under the input viewing angle. D-SSIM Continuously updated. Wherein, based on the first parameter updated in the i-th iteration of the first network model, the parameters of the first network model are updated to obtain an updated first network model, wherein in the i-th iteration update, the loss between the first true value image and the rendered image is less than or equal to the first preset threshold, the first true value image is any image in the training image set, the rendered image is an image obtained based on a three-dimensional Gaussian radiation field, and the rendered image has the same viewing angle as the first true value image, the three-dimensional Gaussian radiation field is generated by the first network model updated in the i-th iteration based on the first true value image, and i is a positive integer. The loss between the first true value image and the rendered image includes L, which can be calculated based on L1 and L D-SSIM Determine and satisfy the following relationship:

[0054] L=(1-λ)L1+λL D-SSIM ;

[0055] L1=||cc gt ||1;

[0056] L D-SSIM =1-SSIM(c,c gt );

[0057] SSIM(x, y) = (2μ x μy )(2σ xy ) / (μ 2 x +μ 2 y )(σ 2 x +σ 2 y );

[0058] Among them, c and c gt are the RGB values ​​of the rendered image and the RGB values ​​of the first truth image respectively; in SSIM, μ x and μ y denote the means of x and y, σ x and σ y represent the standard deviation of x and y respectively, σ xy represents the covariance of x and y, where x and y represent the pixel position of either RGB respectively.

[0059] In this way, the image set is targetedly processed based on the pre-trained first network model to generate a three-dimensional Gaussian radiation field, which comprehensively considers the feature information in the view images corresponding to each object at different perspectives, and continuously updates and iterates the first network model according to the loss between the first true image and the rendered image, thereby improving the accuracy of the generated three-dimensional Gaussian radiation field.

[0060] In some embodiments, the step 102 of spatially dividing the three-dimensional Gaussian radiation field based on the target grid scale to obtain a plurality of grid spaces includes:

[0061] Determining a target grid scale according to the distribution of a mean vector of a three-dimensional Gaussian in the three-dimensional Gaussian radiation field in space;

[0062] The three-dimensional Gaussian radiation field is divided into N×N grids according to the target grid scale to obtain N 3 grid spaces, wherein N is a positive integer, and the number of the mean vectors contained in each grid space is within a preset number range.

[0063] In this implementation, it is considered that when the three-dimensional Gaussian radiation field is spatially divided, if the selected grid scale is too large, it is easy to cause multiple objects to exist in the same grid space, reducing the accuracy of subsequent feature segmentation; if the selected grid scale is too small, it is easy for the same object to exist in multiple grid spaces at the same time, resulting in a large amount of subsequent calculations. Based on this, the target grid scale can be determined according to the distribution of the mean vector of the three-dimensional Gaussian in the three-dimensional Gaussian radiation field in space, so that the number of mean vectors contained in each grid space is within a preset range. For example, the average number of mean vectors contained in each grid space can be 5. According to the target grid scale, the three-dimensional Gaussian radiation field is divided into N×N grids, and N 3 Grid space.

[0064] Taking the mean vector of M three-dimensional Gaussians in each grid space as an example (M=5), in the related art, the number of feature vectors that need to be stored for feature segmentation of the three-dimensional Gaussian radiation field is N 3 In the example provided by the present invention, the three-dimensional Gaussian radiation field is divided into N×N grids according to the target grid scale to obtain N 3 grid spaces, where N 3 The number of mesh vertices in a mesh space is (N+1) 3 In the subsequent feature segmentation process of the three-dimensional Gaussian radiation field, different objects can be segmented based on the feature vectors of the grid vertices in each grid space, so the number of feature vectors that need to be stored is (N+1) 3 (N+1) 3 With N 3 There is the following relationship between ×M:

[0065]

[0066] From the above formula, we can know that the amount of data to be processed in this example can be reduced to 20% of that in the related art, and the more the number of mean vectors M containing three-dimensional Gaussians in each grid space, the greater the number of N 3 The grid vertices of adjacent grid spaces in a grid space are shared with each other, so that the shared grid vertices in adjacent grid spaces use the same feature vector, which reduces the amount of data to be processed and has a better effect, thereby improving the efficiency of three-dimensional scene segmentation.

[0067] In some embodiments, the step 103, performing feature segmentation on the three-dimensional Gaussian radiation field based on the feature vectors of the grid vertices of each grid space, includes:

[0068] Determine the segmentation feature vector corresponding to the first mean vector according to the feature vectors of the grid vertices of the first grid space, wherein the first grid space is the N 3any one of the grid spaces, the first mean vector being any one of the mean vectors of the preset number range contained in the first grid space;

[0069] Determining a three-dimensional segmentation label for each object in the three-dimensional Gaussian radiation field according to each of the segmentation feature vectors;

[0070] The three-dimensional Gaussian radiation field is feature segmented according to the three-dimensional segmentation label.

[0071] In this embodiment, N 3 Each grid space in the grid space corresponds to 8 grid vertices, and each grid vertex can be assigned a feature vector of multiple dimensions (for example, 16 dimensions). The segmentation feature vector determined by the feature vector of the grid vertex can also describe the center point (i.e., mean) of each Gaussian distribution, the expansion direction and range (i.e., covariance matrix) of the Gaussian distribution, the height or intensity of the Gaussian distribution, and the color information in the corresponding Gaussian area represented by RGB values ​​or other color space values.

[0072] N 3 Taking any first grid space in the grid spaces as an example, the first grid space may contain a preset number of mean vectors, for example, the first grid space may contain 5 mean vectors, then the segmentation feature vectors corresponding to the 5 mean vectors in the first grid space may be determined based on the feature vectors of the 8 grid vertices in the first grid space. Similarly, according to (N+1) 3 The eigenvectors corresponding to the mesh vertices can determine N 3 The mean vector in each grid space in the grid space corresponds to the segmentation feature vector. Then, according to each segmentation feature vector, the three-dimensional segmentation label of each object in the three-dimensional Gaussian radiation field is determined. The three-dimensional segmentation label is used to indicate the three-dimensional Gaussian with the same attribute; objects with the same three-dimensional segmentation label are determined as the same category, thereby realizing the feature segmentation of the three-dimensional Gaussian radiation field.

[0073] In this way, in the process of feature segmentation of the three-dimensional Gaussian radiation field, different objects can be segmented based on the feature vectors of the grid vertices in each grid space. Compared with traditional segmentation methods, the amount of data that needs to be processed is reduced and the efficiency of three-dimensional scene segmentation is improved.

[0074] Optionally, determining the segmentation feature vector corresponding to the first mean vector according to the feature vectors of the mesh vertices in the first mesh space includes:

[0075] Trilinear interpolation is performed on the first mean vector according to the feature vectors of the mesh vertices of the first mesh space to obtain a segmentation feature vector corresponding to the first mean vector.

[0076] In this example, feature vectors for scene segmentation are assigned only at the mesh vertices, rather than directly assigning feature vectors to the mean of each three-dimensional Gaussian in space. The feature vector corresponding to the mean vector of the three-dimensional Gaussian is obtained by trilinear interpolation of its mean and the feature vector of the mesh vertex where it is located. In this way, there is no need to assign and store feature vectors for each three-dimensional Gaussian, only feature vectors for mesh vertices are assigned and stored, thereby reducing the number of feature vectors, thereby reducing the amount of data that needs to be processed and reducing the occupation of resources such as video memory.

[0077] Specifically, the feature segmentation of the three-dimensional Gaussian radiation field can be performed based on the pre-trained second network model. The training process of the second network model can be described as follows:

[0078] After obtaining the pre-trained first network model, the three-dimensional Gaussian radiation field of the high-fidelity image under any new perspective output by the first network model can be further spatially divided. According to the position distribution of the three-dimensional Gaussian mean vector in space, the spatial grid of the target grid scale is selected for division, so that the average number of three-dimensional Gaussians contained in the spatial grid is 5. Each grid vertex is assigned a 16-dimensional feature vector f. The segmentation feature vector f corresponding to the mean vector of any three-dimensional Gaussian inside the grid is g It can be obtained by trilinear interpolation of the eigenvectors of the grid vertices in the spatial grid. Then, the 16 dimensions of the eigenvector are extended and the alpha combination is performed in the order of the three-dimensional Gaussians in the space from front to back to obtain the eigenvector of each pixel in the image:

[0079]

[0080] Where i, j are the pixel indices of the image in the training image set, i∈[0,H], j∈[0,W], H and W are the height and width of the image in the training image set, and α is the transparency of the three-dimensional Gaussian. Finally, a feature map for segmentation with the same length and width as the input image but with 16 channels is obtained. To achieve segmentation, the 16-dimensional feature vector at the pixel level After a linear layer and softmax, the probability value of the classification is obtained, and the number of output channels is 256, corresponding to the number of segmented categories. The category label value corresponding to the highest classification score is taken as the image grayscale value and the true value segmentation label grayscale image to calculate the cross entropy loss, and the feature vector including the grid encoding and the linear layer parameters are optimized by gradient descent. The loss function is:

[0081]

[0082] Among them, x ij,c is the probability value corresponding to category c at the pixel position (i, j), y ij,cis the corresponding true value segmentation label. Based on the second parameter of the second network model updated in the jth iteration, the parameters of the second network model are updated to obtain an updated second network model, wherein the cross entropy loss between the first grayscale image and the second grayscale image in the jth iteration update is less than or equal to the second preset threshold, the first grayscale image is a grayscale image corresponding to any image in the training image set, the second grayscale image is generated by the second network model updated in the jth iteration based on the three-dimensional Gaussian radiation field, and the first grayscale image and the second grayscale image have the same viewing angle, and j is a positive integer. In this way, the training of the second network model for feature segmentation of the three-dimensional Gaussian radiation field is realized.

[0083] The segmentation feature vector of each 3D Gaussian in the grid space can be obtained by trilinear interpolation of the feature vectors of the grid partition vertices. The segmentation feature vectors of all interpolated 3D Gaussians are passed through the trained linear layer and softmax to obtain the probability values ​​of 256 categories. The classification threshold is set to 0.4, and all segmentation categories are traversed. The 3D Gaussians that exceed the threshold are marked as the current segmentation category, and finally the 3D scene segmentation is realized. The mean point of the 3D Gaussian can be displayed as a 3D segmented point cloud.

[0084] In this way, by performing feature segmentation processing on the three-dimensional Gaussian radiation field based on the pre-trained second network model, different objects can be segmented based on the feature vectors of the grid vertices in each grid space, and the feature information in the view image corresponding to each object at different perspectives is comprehensively considered. The second network model is continuously updated and iterated according to the cross entropy loss between the first grayscale image and the second grayscale image, thereby improving the accuracy of feature segmentation of the three-dimensional Gaussian radiation field.

[0085] See also Figure 2 , Figure 2 is a flow chart of a model training method provided by an embodiment of the present invention, such as Figure 2 As shown, the method comprises the following steps:

[0086] Step 201, iteratively train the first module and the second module of the initial model based on a training image set, wherein the training image set includes images corresponding to multiple objects at different perspectives, and the pixel positions in the images corresponding to the same object among the multiple objects at different perspectives are marked by two-dimensional segmentation labels, the first module is used to generate a three-dimensional Gaussian radiation field, and the second module is used to perform feature segmentation on the three-dimensional Gaussian radiation field.

[0087] The first module may be the first network model in the above embodiment, and the second module may be the second network model in the above embodiment. The training image set may be a series of image data collected by a camera in a real space (e.g., a bedroom, a football field, a park, etc.). The training image set is preprocessed so that the training image set includes view images corresponding to multiple objects at different viewing angles, and the pixel positions in the view images corresponding to the same object in the multiple objects at different viewing angles are marked by two-dimensional segmentation labels. In addition, the two-dimensional segmentation labels are optimized using the morphological corrosion and expansion operation in image processing to eliminate errors and noise in the segmentation labels.

[0088] In an embodiment of the present invention, by marking the pixel positions in the images corresponding to the same object in multiple objects at different viewing angles with two-dimensional segmentation labels, a more accurate three-dimensional scene segmentation result can be obtained. By gridding the space and interpolating the three-dimensional Gaussian segmentation feature vector, the grid vertices of adjacent grid spaces in multiple grid spaces are shared with each other. In other words, the shared grid vertices in adjacent grid spaces can use the same feature vector. In this way, in the process of training the feature segmentation of the three-dimensional Gaussian radiation field, segmentation training can be performed on different objects based on the feature vectors of the grid vertices in each grid space, which reduces the amount of data to be processed compared to the traditional segmentation training method, thereby improving the training efficiency of the training model.

[0089] Optionally, the iterative training of the first module and the second module of the initial model based on the training image set includes:

[0090] Based on the first parameter updated by the first module in the i-th iteration, the parameter of the first module is updated to obtain an updated first module, wherein the loss between the first true value image and the rendered image in the i-th iteration update is less than or equal to a first preset threshold, the first true value image is any image in the training image set, the rendered image is an image obtained based on a three-dimensional Gaussian radiation field, and the rendered image has the same viewing angle as the first true value image, the three-dimensional Gaussian radiation field is generated by the first module updated in the i-th iteration based on the first true value image, and i is a positive integer;

[0091] and / or,

[0092] Based on the second parameter updated by the second module in the jth iteration, the parameter of the second module is updated to obtain an updated second module, wherein the cross entropy loss between the first grayscale image and the second grayscale image in the jth iteration update is less than or equal to a second preset threshold, the first grayscale image is a grayscale image corresponding to any image in the training image set, the second grayscale image is generated by the second module updated in the jth iteration based on the three-dimensional Gaussian radiation field, and the first grayscale image and the second grayscale image have the same viewing angle, and j is a positive integer.

[0093] In this way, the image set is targetedly processed through the pre-trained first module to generate a three-dimensional Gaussian radiation field, which comprehensively considers the feature information in the view images corresponding to each object at different perspectives, and continuously updates and iterates the first network model according to the loss between the first true image and the rendered image, thereby improving the accuracy of the generated three-dimensional Gaussian radiation field.

[0094] And / or, by performing feature segmentation processing on the three-dimensional Gaussian radiation field through a pre-trained second module, different objects can be segmented based on the feature vectors of the grid vertices in each grid space, and the feature information in the view image corresponding to each object at different perspectives is comprehensively considered. The second network model is continuously updated and iterated according to the cross entropy loss between the first grayscale image and the second grayscale image, thereby improving the accuracy of feature segmentation of the three-dimensional Gaussian radiation field.

[0095] See also Figure 3 , Figure 3 is a schematic diagram of the structure of an image processing device provided by an embodiment of the present invention. Figure 3 As shown, the image processing device 300 includes:

[0096] A processing module 301 is used to perform target processing on an image set to generate a three-dimensional Gaussian radiation field, wherein the image set includes images corresponding to a first object at different viewing angles, and the target processing is used to associate feature labels in the images corresponding to the first object at different viewing angles;

[0097] A division module 302 is used to spatially divide the three-dimensional Gaussian radiation field based on a target grid scale to obtain a plurality of grid spaces;

[0098] The segmentation module 303 is used to perform feature segmentation on the three-dimensional Gaussian radiation field based on the feature vectors of the grid vertices in each grid space.

[0099] Optionally, the division module 302 is further configured to:

[0100] Determining a target grid scale according to the distribution of a mean vector of a three-dimensional Gaussian in the three-dimensional Gaussian radiation field in space;

[0101] The three-dimensional Gaussian radiation field is divided into N×N grids according to the target grid scale to obtain N 3 grid spaces, wherein N is a positive integer, and the number of the mean vectors contained in each grid space is within a preset number range.

[0102] Optionally, the segmentation module 303 is further used for:

[0103] Determine the segmentation feature vector corresponding to the first mean vector according to the feature vectors of the grid vertices of the first grid space, wherein the first grid space is the N 3 any one of the grid spaces, the first mean vector being any one of the mean vectors of the preset number range contained in the first grid space;

[0104] Determining a three-dimensional segmentation label for each object in the three-dimensional Gaussian radiation field according to each of the segmentation feature vectors;

[0105] The three-dimensional Gaussian radiation field is feature segmented according to the three-dimensional segmentation label.

[0106] Optionally, determining the segmentation feature vector corresponding to the first mean vector according to the feature vectors of the mesh vertices in the first mesh space includes:

[0107] Trilinear interpolation is performed on the first mean vector according to the feature vectors of the mesh vertices of the first mesh space to obtain a segmentation feature vector corresponding to the first mean vector.

[0108] Optionally, the processing module 301 is further used for:

[0109] Performing feature recognition processing on the image set to obtain feature labels in images corresponding to multiple objects in the image set at different viewing angles, wherein the first object is any one of the multiple objects;

[0110] Associating feature labels whose matching degree between the feature labels is greater than or equal to a preset matching threshold, to obtain two-dimensional segmentation labels corresponding to the multiple objects respectively, wherein the two-dimensional segmentation labels are used to indicate pixel positions in images corresponding to the same object among the multiple objects at different viewing angles;

[0111] The three-dimensional Gaussian radiation fields corresponding to the multiple objects are generated according to the two-dimensional segmentation labels respectively corresponding to the multiple objects.

[0112] The image processing device 300 is capable of implementing each process of each embodiment of the above-mentioned image processing method. The technical features correspond one to one and can achieve the same technical effect. To avoid repetition, they will not be described again here.

[0113] See also Figure 4 , Figure 4 is a schematic diagram of the structure of a model training device provided by an embodiment of the present invention, such as Figure 4 As shown, the model training device 400 includes:

[0114] The training module 401 is used to iteratively train the first module and the second module of the initial model based on a training image set, wherein the training image set includes images corresponding to multiple objects at different perspectives, and the pixel positions in the images corresponding to the same object among the multiple objects at different perspectives are marked by two-dimensional segmentation labels, the first module is used to generate a three-dimensional Gaussian radiation field, and the second module is used to perform feature segmentation on the three-dimensional Gaussian radiation field.

[0115] Optionally, the training module 401 is further used for:

[0116] Based on the first parameter updated by the first module in the i-th iteration, the parameter of the first module is updated to obtain an updated first module, wherein the loss between the first true value image and the rendered image in the i-th iteration update is less than or equal to a first preset threshold, the first true value image is any image in the training image set, the rendered image is an image obtained based on a three-dimensional Gaussian radiation field, and the rendered image has the same viewing angle as the first true value image, the three-dimensional Gaussian radiation field is generated by the first module updated in the i-th iteration based on the first true value image, and i is a positive integer;

[0117] and / or,

[0118] Based on the second parameter updated by the second module in the jth iteration, the parameter of the second module is updated to obtain an updated second module, wherein the cross entropy loss between the first grayscale image and the second grayscale image in the jth iteration update is less than or equal to a second preset threshold, the first grayscale image is a grayscale image corresponding to any image in the training image set, the second grayscale image is generated by the second module updated in the jth iteration based on the three-dimensional Gaussian radiation field, and the first grayscale image and the second grayscale image have the same viewing angle, and j is a positive integer.

[0119] The model training device 400 is capable of implementing each process of each embodiment of the above-mentioned model training method. The technical features correspond one to one and can achieve the same technical effect. To avoid repetition, they will not be described here.

[0120] An embodiment of the present invention further provides an electronic device, including:

[0121] at least one processor; and

[0122] a memory communicatively connected to the at least one processor; wherein,

[0123] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the various processes of the above-mentioned image processing method embodiment, or so that the at least one processor can execute the various processes of the above-mentioned model training method embodiment, and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0124] The embodiment of the present invention further provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, each process of the above-mentioned image processing method embodiment is implemented, or when the computer program is executed by a processor, each process of the above-mentioned model training method embodiment is implemented, and the same technical effect can be achieved, so it will not be repeated here to avoid repetition. The computer-readable storage medium is, for example, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0125] The embodiments of the present application also provide a computer program product, including computer instructions. When the computer instructions are executed by a processor, the various processes of the above-mentioned image processing method embodiment are implemented, or when the computer instructions are executed by a processor, the various processes of the above-mentioned model training method embodiment are implemented, and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0126] It should be noted that, in this article, the terms "comprise", "include" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the presence of other identical elements in the process, method, article or device including the element. In addition, it should be pointed out that the scope of the method and device in the embodiment of the present invention is not limited to performing functions in the order discussed, and may also include performing functions in a substantially simultaneous manner or in reverse order according to the functions involved, for example, the described method may be performed in an order different from that described, and various steps may also be added, omitted, or combined. In addition, the features described with reference to certain examples may be combined in other examples.

[0127] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, a magnetic disk, or an optical disk), and includes a number of instructions for enabling a terminal (which can be a mobile phone, a computer, a server, an air conditioner, or a network device, etc.) to execute the methods described in each embodiment of the present invention.

[0128] The embodiments of the present invention are described above in conjunction with the accompanying drawings, but the present invention is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the enlightenment of the present invention, ordinary technicians in this field can also make many forms without departing from the scope of protection of the present invention and the claims, all of which are within the protection of the present invention.

Claims

1. An image processing method, It is characterized in that The method comprises: Performing target processing on an image set to generate a three-dimensional Gaussian radiation field, the image set including images corresponding to a first object at different viewing angles, the target processing being used to associate feature labels in the images corresponding to the first object at different viewing angles; Based on the target grid scale, the three-dimensional Gaussian radiation field is spatially divided to obtain a plurality of grid spaces; Based on the feature vectors of the grid vertices of each grid space, feature segmentation is performed on the three-dimensional Gaussian radiation field; The three-dimensional Gaussian radiation field is spatially divided based on the target grid scale to obtain a plurality of grid spaces, including: Determining a target grid scale according to the distribution of a mean vector of a three-dimensional Gaussian in the three-dimensional Gaussian radiation field in space; The three-dimensional Gaussian radiation field is divided into N×N grids according to the target grid scale to obtain N 3 grid spaces, wherein N is a positive integer, and the number of the mean vectors contained in each grid space is within a preset number range; The feature segmentation of the three-dimensional Gaussian radiation field based on the feature vectors of the grid vertices of each grid space includes: Determine the segmentation feature vector corresponding to the first mean vector according to the feature vectors of the grid vertices of the first grid space, wherein the first grid space is the N 3 any one of the grid spaces, the first mean vector being any one of the mean vectors of the preset number range contained in the first grid space; Determining a three-dimensional segmentation label for each object in the three-dimensional Gaussian radiation field according to each of the segmentation feature vectors; Performing feature segmentation on the three-dimensional Gaussian radiation field according to the three-dimensional segmentation label; The step of determining the segmentation feature vector corresponding to the first mean vector according to the feature vectors of the mesh vertices in the first mesh space includes: Trilinear interpolation is performed on the first mean vector according to the feature vectors of the mesh vertices of the first mesh space to obtain a segmentation feature vector corresponding to the first mean vector.

2. The method according to claim 1, It is characterized in that The target processing of the image set to generate a three-dimensional Gaussian radiation field includes: Performing feature recognition processing on the image set to obtain feature labels in images corresponding to multiple objects in the image set at different viewing angles, wherein the first object is any one of the multiple objects; Associating feature labels whose matching degree between the feature labels is greater than or equal to a preset matching threshold, to obtain two-dimensional segmentation labels corresponding to the multiple objects respectively, wherein the two-dimensional segmentation labels are used to indicate pixel positions in images corresponding to the same object among the multiple objects at different viewing angles; The three-dimensional Gaussian radiation fields corresponding to the multiple objects are generated according to the two-dimensional segmentation labels respectively corresponding to the multiple objects.

3. A model training method, It is characterized in that The method comprises: The first module and the second module of the initial model are iteratively trained based on a training image set, wherein the training image set includes images corresponding to multiple objects at different viewing angles, and the pixel positions in the images corresponding to the same object among the multiple objects at different viewing angles are marked by two-dimensional segmentation labels, the first module is used to generate the three-dimensional Gaussian radiation field described in any one of claims 1 to 2, and the second module is used to perform feature segmentation on the three-dimensional Gaussian radiation field.

4. The method according to claim 3, It is characterized in that The iterative training of the first module and the second module of the initial model based on the training image set comprises: Based on the first parameter updated by the first module in the i-th iteration, the parameter of the first module is updated to obtain an updated first module, wherein the loss between the first true value image and the rendered image in the i-th iteration update is less than or equal to a first preset threshold, the first true value image is any image in the training image set, the rendered image is an image obtained based on a three-dimensional Gaussian radiation field, and the rendered image has the same viewing angle as the first true value image, the three-dimensional Gaussian radiation field is generated by the first module updated in the i-th iteration based on the first true value image, and i is a positive integer; and / or, Based on the second parameter updated by the second module in the jth iteration, the parameter of the second module is updated to obtain an updated second module, wherein the cross entropy loss between the first grayscale image and the second grayscale image in the jth iteration update is less than or equal to a second preset threshold, the first grayscale image is a grayscale image corresponding to any image in the training image set, the second grayscale image is generated by the second module updated in the jth iteration based on the three-dimensional Gaussian radiation field, and the first grayscale image and the second grayscale image have the same viewing angle, and j is a positive integer.

5. An image processing device, It is characterized in that The device comprises: a processing module, configured to perform target processing on an image set to generate a three-dimensional Gaussian radiation field, wherein the image set includes images corresponding to a first object at different viewing angles, and the target processing is configured to associate feature labels in images corresponding to the first object at different viewing angles; A partitioning module, used for spatially partitioning the three-dimensional Gaussian radiation field based on a target grid scale to obtain a plurality of grid spaces; A segmentation module, used for performing feature segmentation on the three-dimensional Gaussian radiation field based on feature vectors of grid vertices in each grid space; The partitioning module is also used for: Determining a target grid scale according to the distribution of a mean vector of a three-dimensional Gaussian in the three-dimensional Gaussian radiation field in space; The three-dimensional Gaussian radiation field is divided into N×N grids according to the target grid scale to obtain N 3 grid spaces, wherein N is a positive integer, and the number of the mean vectors contained in each grid space is within a preset number range; The segmentation module is also used for: Determine the segmentation feature vector corresponding to the first mean vector according to the feature vectors of the grid vertices of the first grid space, wherein the first grid space is the N 3 any one of the grid spaces, the first mean vector being any one of the mean vectors of the preset number range contained in the first grid space; Determining a three-dimensional segmentation label for each object in the three-dimensional Gaussian radiation field according to each of the segmentation feature vectors; Performing feature segmentation on the three-dimensional Gaussian radiation field according to the three-dimensional segmentation label; The step of determining the segmentation feature vector corresponding to the first mean vector according to the feature vectors of the mesh vertices in the first mesh space includes: Trilinear interpolation is performed on the first mean vector according to the feature vectors of the mesh vertices of the first mesh space to obtain a segmentation feature vector corresponding to the first mean vector.

6. A model training device, It is characterized in that The device comprises: A training module, used for iteratively training a first module and a second module of an initial model based on a training image set, wherein the training image set includes images corresponding to multiple objects at different viewing angles, and the pixel positions in the images corresponding to the same object among the multiple objects at different viewing angles are marked by two-dimensional segmentation labels, the first module is used to generate a three-dimensional Gaussian radiation field as described in any one of claims 1 to 2, and the second module is used to perform feature segmentation on the three-dimensional Gaussian radiation field.

7. An electronic device, It is characterized in that include: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the steps of the method as described in any one of claims 1 to 2, or to enable the at least one processor to perform the steps of the method as described in any one of claims 3 to 4.

8. A computer-readable storage medium, It is characterized in that The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the computer program implements the steps of the method according to any one of claims 1 to 2, or implements the steps of the method according to any one of claims 3 to 4.

9. A computer program product, It is characterized in that The method comprises computer instructions, which, when executed by a processor, implement the steps of the method according to any one of claims 1 to 2, or implement the steps of the method according to any one of claims 3 to 4.

Images