Linear discriminative hashing coding modeling method and device based on front-back separation

Abstract

The application discloses a linear discriminant hash coding modeling method based on front-back separation, and belongs to the technical field of image processing. The method comprises the following steps: obtaining a data set of a target scene, performing segmentation on each frame of image in the data set based on an instance segmentation network to obtain a plurality of segmented object objects and background objects, and establishing a background neural radiation field model based on the background objects; performing cross-frame association on the object objects in different frames based on semantic consistency and spatial consistency, inputting the same object objects in different frames into the same local multi-layer perception network to establish a neural radiation field model, obtaining a plurality of object object neural radiation field models, performing hash coding on the background neural radiation field model and the plurality of object object neural radiation field models based on linear discriminant analysis, modeling the hash code corresponding to the background and the hash code corresponding to each object object based on the background, and obtaining a target neural radiation field model, so that the speed and accuracy of scene image modeling are improved.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of image processing, and particularly relates to a linear discriminative hash coding modeling method and device based on front-back separation. BACKGROUND

[0002] With the rapid development of NeRF (Neural Radiance Fields), it has shown significant advantages in three-dimensional reconstruction and scene representation in the fields of autonomous driving, virtual reality, augmented reality, etc., especially in complex geometric structure modeling and high-fidelity view synthesis, it can generate three-dimensional scenes with rich details and realistic. However, in the modeling of dynamic scenes such as autonomous driving, how to realize dynamic target perception and efficient reconstruction of large-scale continuous scenes is still a core challenge that restricts the further application of neural radiation field technology.

[0003] In the actual neural radiation field modeling process, the use of hash coding can greatly improve the speed and quality of modeling, but it is easy to cause hash collision problems. At the same time, the principle of neural radiation field leads to the size limitation of the model, and the scene is too large to make the reconstruction quality drop sharply, and even may appear in the training process Scene invisible situation, unable to meet the actual continuous scene modeling requirements, in addition, the classic volume rendering formula used in neural radiation field can only capture static scenes, and can only focus on objects that do not change with time. However, in actual application scenarios, dynamic foreground in neural radiation field modeling is an important factor that needs to be considered.

[0004] In addition, existing methods are mostly limited to static scene assumptions and do not consider the separation modeling mechanism of dynamic foreground and static background, resulting in the coupled expression of dynamic targets such as pedestrians and vehicles and the background in autonomous driving scenes, causing training divergence and rendering distortion. The existing method has the problems of slow modeling speed, low modeling accuracy, easy to produce artifacts, and model training difficult to converge in the process of dynamic neural radiation field modeling. SUMMARY

[0005] The present application aims to at least solve one of the technical problems existing in the prior art. To this end, the present application proposes a linear discriminative hash coding modeling method and device based on front-back separation, which improves the speed and accuracy of scene image modeling.

[0006] In a first aspect, the present application provides a linear discriminative hash coding modeling method based on front-back separation, the method comprising:

[0007] obtaining a data set of a target scene, the data set comprising a plurality of image data;

[0008] The instance segmentation network is used for segmenting each frame of image in the data set to obtain a plurality of segmented object objects and background objects, and a background neural radiance field model is established based on the background objects;

[0009] Each object object in each frame of image is labeled to obtain a plurality of object masks, and each object object corresponds to an object mask;

[0010] The object objects in different frames are associated based on semantic consistency and spatial consistency, and the same object object in different frames is input into the same local multi-layer perception network to establish a neural radiance field model, thereby obtaining a plurality of object object neural radiance field models, each object object corresponding to a neural radiance field model;

[0011] The background neural radiance field model and the plurality of object object neural radiance field models are hashed based on linear discriminant analysis to obtain a hash code corresponding to the background and a hash code corresponding to each object object;

[0012] The hash code corresponding to the background and the hash code corresponding to each object object are modeled to obtain a target neural radiance field model, and the target neural radiance field model includes the background and all object objects.

[0013] According to an embodiment of the present application, the instance segmentation network is used for segmenting each frame of image in the data set to obtain a plurality of segmented object objects and background objects, including:

[0014] Each frame of image in the data set is input into the instance segmentation network to obtain a plurality of category score bounding boxes;

[0015] The plurality of category score bounding boxes are de-duplicated based on a non-maximum suppression algorithm to obtain a plurality of segmented object objects and background objects.

[0016] According to an embodiment of the present application, the non-maximum suppression algorithm is used for de-duplicating the plurality of category score bounding boxes to obtain a plurality of segmented object objects and background objects, including:

[0017] The first n score bounding boxes of each category are sorted according to the scores, the intersection over union between the same category bounding boxes is calculated, and an intersection over union matrix is obtained based on the intersection over union between the same category bounding boxes;

[0018] Based on the intersection over union matrix of each category, the score bounding box with the highest score result is retained, and the segmented plurality of object objects and background objects are obtained in combination with the mask coefficient and the object mask.

[0019] According to an embodiment of the present application, the object objects in different frames are associated based on semantic consistency and spatial consistency, including:

[0020] For each object object in each frame of image, it is judged whether the semantic class of the object object in the current frame and the object object in the next frame is same, if same, it is judged the closeness of the object object in the current frame and the object object in the next frame in space, if not same, the object object in the current frame and the object object in the next frame are different object objects;

[0021] When the closeness is greater than or equal to the preset threshold, the object object in the current frame and the object object in the next frame are the same object object;

[0022] When the closeness is less than the preset threshold, the object object in the current frame and the object object in the next frame are different object objects.

[0023] According to an embodiment of the present application, the hash coding of the background neural radiation field model and the plurality of object object neural radiation field models based on linear discriminant analysis comprises:

[0024] Based on the background neural radiation field model and the plurality of object object neural radiation field models, a target function is constructed respectively, to obtain a background target function and a plurality of object object target functions;

[0025] Based on the background target function and the plurality of object object target functions, the background neural radiation field model and the plurality of object object neural radiation field models are hash coded.

[0026] According to an embodiment of the present application, the expression of the target function is as follows:

[0027] y(x)=sign(H T x)

[0028] Wherein, y(x) is a hash code, x is source data, and H is a mapping matrix.

[0029] According to an embodiment of the present application, the instance segmentation network comprises a backbone network, a prediction network, a prototype network and a mask network, and the backbone network comprises a feature pyramid network and a full convolution network.

[0030] In a second aspect, the present application provides a linear discriminant hash coding modeling device based on front-back separation, the device comprises:

[0031] The acquisition module is used for acquiring a data set of a target scene, and the data set comprises a plurality of frame image data;

[0032] The segmentation module is used for segmenting each frame of image in the data set based on an instance segmentation network, to obtain a plurality of segmented object objects and background objects, and a background neural radiation field model is established based on the background objects;

[0033] A marking module is configured to mark each object in each frame of images to obtain a plurality of object masks, each object corresponding to an object mask;

[0034] An association module is configured to associate the objects in different frames based on semantic consistency and spatial consistency, input the same object in different frames into the same local multi-layer perception network to establish a neural radiance field model, and obtain a plurality of object neural radiance field models, each object corresponding to a neural radiance field model;

[0035] An encoding module is configured to perform hash coding on the background neural radiance field model and the plurality of object neural radiance field models based on linear discriminant analysis to obtain a hash code corresponding to the background and a hash code corresponding to each object.

[0036] A modeling module is configured to model based on the hash code corresponding to the background and the hash code corresponding to each object to obtain a target neural radiance field model, the target neural radiance field model including the background and all the objects.

[0037] In a third aspect, the present application provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, and the processor implements the front-back separation-based linear discriminant hash coding modeling method according to the first aspect when executing the computer program.

[0038] In a fourth aspect, the present application provides a non-transitory computer-readable storage medium having a computer program stored thereon, and the computer program is executable by a processor to implement the front-back separation-based linear discriminant hash coding modeling method according to the first aspect.

[0039] In a fifth aspect, the present application provides a chip, including a processor and a communication interface, the communication interface and the processor are coupled, and the processor is configured to run a program or instructions to implement the front-back separation-based linear discriminant hash coding modeling method according to the first aspect.

[0040] In a sixth aspect, the present application provides a computer program product, including a computer program, and the computer program is executable by a processor to implement the front-back separation-based linear discriminant hash coding modeling method according to the first aspect.

[0041] Additional aspects and advantages of the present application will be in part apparent and in part pointed out hereinafter.

[0042] The front-back separation-based linear discriminant hash coding modeling method provided by the present application has the following beneficial effects compared with the prior art:

[0043] (1) The present application effectively reduces the computational complexity and improves the accuracy and efficiency of 3D scene reconstruction and object recognition by performing instance segmentation on each frame of image in the target scene data set, combining a background neural radiance field model and multiple object neural radiance field models, and using linear discriminant analysis for hash coding. By segmenting the background and foreground target objects and associating them between different frames, respectively modeling the neural radiance field, the occurrence of artifacts and non-convergence of model training during the neural radiance field modeling process can be reduced. The use of linear discriminant hash coding effectively reduces the hash collision problem during the hash coding modeling process, improving the stability and reliability of the neural radiance field modeling.

[0044] (2) The present application can better segment object and background objects by performing instance segmentation on each frame of image in the data set and combining non-maximum suppression algorithm to remove duplicate detection boxes, improving the accuracy of image segmentation results, reducing the influence of redundant detection boxes, and improving the stability and reliability of subsequent neural radiance field modeling, and improving the training efficiency of the neural radiance field model.

[0045] (3) The present application can effectively improve the discrimination of background and object by constructing the objective function of the background neural radiance field model and multiple object neural radiance field models respectively, and performing hash coding based on the objective function. By using linear discriminant analysis for hash coding optimization of the original neural radiance field position coding, the characteristics of the data are fully considered during the coding process. By extracting key features and mapping them to hash codes, the hash codes can more accurately reflect the characteristics of the original data, effectively reducing the hash collision problem during the neural radiance field modeling process, and improving the modeling efficiency. BRIEF DESCRIPTION OF DRAWINGS

[0046] The above and / or additional aspects and advantages of the present application will become apparent and more readily appreciated from the following description of the embodiments, taken in conjunction with the accompanying drawings, in which:

[0047] Figure 1 is one of the flowcharts of the linear discriminant hash coding modeling method based on front-back separation provided by the embodiments of the present application;

[0048] Figure 2 is a structural diagram of the YOLACT network provided by the embodiments of the present application;

[0049] Figure 3 is a structural diagram of the prediction network;

[0050] Figure 4 is the second flowchart of the linear discriminant hash coding modeling method based on front-back separation provided by the embodiments of the present application;

[0051] Figure 5is a flowchart of a hash coding method based on linear discriminant analysis provided by an embodiment of the present application.

[0052] Figure 6 is a structural diagram of a linear discriminant hash coding modeling device based on front-back separation provided by an embodiment of the present application.

[0053] Figure 7 is a structural diagram of an electronic device provided by an embodiment of the present application. DETAILED DESCRIPTION

[0054] The technical solutions in the embodiments of the present application will be clearly described below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are some but not all of the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by a person of ordinary skill in the art belong to the scope of protection of the present application.

[0055] The terms "first", "second", and the like in the specification and claims of the present application are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged under appropriate circumstances, so that the embodiments of the present application can be implemented in an order other than those illustrated or described herein, and the objects distinguished by "first", "second", etc. are generally a class, not limited to the number of objects, for example, the first object can be one or more. In addition, "and / or" in the specification and claims means at least one of the connected objects, and the character " / ", generally represents a "or" relationship between the front and rear associated objects.

[0056] The linear discriminant hash coding modeling method based on front-back separation, the linear discriminant hash coding modeling device based on front-back separation, the electronic device and the readable storage medium provided by the embodiments of the present application will be described in detail below in combination with the drawings and specific embodiments and their application scenarios.

[0057] The linear discriminant hash coding modeling method based on front-back separation can be applied to a terminal, and can be specifically executed by hardware or software in the terminal.

[0058] The terminal includes, but is not limited to, a portable communication device such as a mobile phone or a tablet computer having a touch-sensitive surface (e.g., a touchscreen display and / or a touchpad). It should also be understood that, in some embodiments, the terminal can not be a portable communication device, but a desktop computer having a touch-sensitive surface (e.g., a touchscreen display and / or a touchpad).

[0059] In the following various embodiments, a terminal including a display and a touch-sensitive surface is described. However, it should be understood that the terminal can include one or more other physical user interface devices such as physical keyboards, mice, and joysticks.

[0060] The execution subject of the front-back separation-based linear discriminant hash coding modeling method provided in the embodiments of the present application can be an electronic device or a functional module or functional entity capable of implementing the front-back separation-based linear discriminant hash coding modeling method in the electronic device. The electronic device mentioned in the embodiments of the present application includes but is not limited to a mobile phone, a tablet computer, a computer, a camera, a wearable device, and the like. The front-back separation-based linear discriminant hash coding modeling method provided in the embodiments of the present application will be described below by taking an electronic device as an execution subject.

[0061] Figure 1 is one of the flowcharts of the front-back separation-based linear discriminant hash coding modeling method provided in the embodiments of the present application, as shown in Figure 1 The front-back separation-based linear discriminant hash coding modeling method includes steps 110, 120, 130, 140, 150, and 160.

[0062] Step 110, obtaining a data set of a target scene, the data set including multiple frames of image data;

[0063] It is easy to understand that the image data of the target scene collected by different cameras at different distances and different time steps is obtained to obtain the data set of the target scene.

[0064] Step 120, segmenting each frame of image in the data set based on an instance segmentation network to obtain multiple segmented object objects and background objects, and establishing a background neural radiance field model based on the background objects;

[0065] It should be noted that each frame of image in the data set includes multiple object objects and background objects. Since the relative positions of each object object and the background object in each frame of image are different, it is necessary to segment each frame of image in the data set to separate the object objects and the background objects in each frame of image.

[0066] In some embodiments, the instance segmentation network includes a backbone network, a prediction network, a prototype network, and a mask network, and the backbone network includes a feature pyramid network and a fully convolutional network.

[0067] For example, the instance segmentation network is a YOLACT network, which can realize fast and end-to-end instance segmentation, and each frame of image in the data set is segmented by the YOLACT network to associate the images between different frames.

[0068] Figure 2 is a structural schematic diagram of the YOLACT network provided in the embodiments of the present application, as shown in Figure 2As shown, the YOLACT network includes a backbone network, a prediction network, a prototype network and a mask network, the backbone network includes a feature pyramid network and a full convolution network, wherein C1-C5 are full convolution networks for extracting features of the image, and P3-P7 are feature pyramid networks for fusing multi-scale information of the image, the multi-scale information is used to generate a prototype mask by the prototype network and a prediction mask by the prediction network, Figure 3 is a structural diagram of the prediction network, as shown in Figure 3 As shown, the prediction network adopts a shared convolution network, and after upsampling, is input into three parallel branches of a category layer, a bounding box layer and a mask layer, each pixel point predicts 3 anchor boxes, and finally predicts (4+c+k) values for each anchor box, wherein 4 represents anchor box coordinate offset, c represents anchor box category number, and k represents prototype mask coefficient.

[0069] Finally, a plurality of segmented object objects and background objects are obtained through the prototype mask and the prediction mask.

[0070] In this embodiment, by combining the backbone network, the prediction network, the prototype network and the mask network to perform instance segmentation on each frame of image, the feature pyramid network and the full convolution network are used for feature extraction, which effectively improves the accuracy and robustness of object segmentation. Through multi-level feature fusion, the background and object objects in the image can be better recognized and separated.

[0071] Step 130, labeling each object object in each frame of image to obtain a plurality of object masks, each object object corresponding to an object mask;

[0072] Step 140, based on semantic consistency and spatial consistency, cross-frame association of object objects in different frames is performed, the same object object in different frames is input into the same local multi-layer perception network to establish a neural radiation field model, and a plurality of object object neural radiation field models are obtained, each object object corresponding to a neural radiation field model;

[0073] It is easy to understand that, in order to ensure the continuity and integrity of the background model, the object objects in different frames are cross-frame associated according to the object masks based on semantic consistency and spatial consistency, the semantic consistency refers to whether the semantics between the object objects are consistent, and the spatial consistency refers to whether the spatial position relationship of the object objects in different frames is consistent.

[0074] Step 150, based on linear discriminant analysis, the background neural radiation field model and the plurality of object object neural radiation field models are hashed coded to obtain a hash code corresponding to the background and a hash code corresponding to each object object;

[0075] Further, three-dimensional data points in the background neural radiance field model and the plurality of object neural radiance field models are acquired, for each three-dimensional data point, according to the category to which it belongs, a corresponding hash code is collected as an original feature vector to form a labeled training data set, then a target function is established through linear discriminant, a projection matrix is solved, and the original hash code is projected to a low-dimensional space through the projection matrix to obtain a hash code corresponding to the background and a hash code corresponding to each object.

[0076] Step 160, modeling based on the hash code corresponding to the background and the hash code corresponding to each object object, obtaining a target neural radiance field model, the target neural radiance field model including the background and all object objects.

[0077] Figure 4 Figure 2 is a flowchart of a linear discriminant hash coding modeling method based on front-back separation provided by an embodiment of the present application, multi-view multi-scale data and depth data are acquired, the multi-view multi-scale data being image data of a target scene collected by different cameras at different distances and different time steps, the depth data being point cloud data corresponding to the image data, a target instance segmentation network is used to separate and associate the image data, then a linear discriminant analysis method is used for hash coding and generating a neural radiance field model for the background and the object object respectively, finally, the hash code corresponding to the background and the hash code corresponding to each object object are taken as input features of the neural radiance field, a multi-layer perception network is used to learn the radiance field (color, density) distribution of the background and each object respectively, and the radiance fields of different components are fused through a spatial mask or a probability weight, so that the model can dynamically select the hash features of the background or the corresponding object according to the three-dimensional coordinates during rendering, and a target neural radiance field model is obtained.

[0078] According to the linear discriminant hash coding modeling method based on front-back separation provided by the embodiment of the present application, each frame of image in the target scene data set is segmented by an instance segmentation network, a background neural radiance field model and a plurality of object neural radiance field models are combined, and a linear discriminant analysis is used for hash coding, which effectively reduces the computational complexity and improves the accuracy and efficiency of 3D scene reconstruction and object recognition, the background and the foreground target object are segmented and associated between different frames, and neural radiance field modeling is performed respectively, which can reduce the occurrence of artifacts and the non-convergence of model training in the process of dynamic target in neural radiance field modeling, the use of linear discriminant hash coding effectively reduces the hash collision problem in the process of hash coding modeling, and improves the stability and reliability of neural radiance field modeling.

[0079] In some embodiments, the instance segmentation network is used to segment each frame of image in the data set to obtain a plurality of segmented object objects and a background object, including:

[0080] Input each frame of image in the data set into the instance segmentation network to obtain a plurality of category score bounding boxes;

[0081] De-duplicate the plurality of category score bounding boxes based on a non-maximum suppression algorithm to obtain a plurality of segmented object objects and background objects.

[0082] It is easy to understand that inputting each frame of image in the data set into the instance segmentation network obtains a plurality of category score bounding boxes, each category represents a different object object, and one category may include one or more score bounding boxes. Therefore, the non-maximum suppression algorithm is used to de-duplicate the plurality of score bounding boxes of each category, so that each object object and background object corresponds to only one score bounding box.

[0083] In some embodiments, the de-duplication of the plurality of category score bounding boxes based on the non-maximum suppression algorithm to obtain the plurality of segmented object objects and background objects comprises:

[0084] Sorting the first n score bounding boxes of each category according to the score, calculating the intersection over union between the same category bounding boxes, and obtaining an intersection over union matrix based on the intersection over union between the same category bounding boxes;

[0085] Based on the intersection over union matrix of each category, the score bounding box with the highest score result is retained, and the mask coefficient and the object mask are combined to obtain the plurality of segmented object objects and background objects.

[0086] For example, the process of the non-maximum suppression algorithm is as follows:

[0087] (1) For the top n results of each category with the highest score (the probability that the target in the bounding box is the current category), arrange them in descending order, and then calculate the intersection over union between each other to obtain a Cxnxn (where C is the total number of categories) matrix N, and each n x n matrix in it is a diagonal matrix.

[0088] (2) For a certain category, remove the bounding boxes that overlap with the higher score results. After filtering out the overlapping bounding boxes, linearly combine the mask coefficient and the object mask, and perform matrix multiplication on the object mask and the mask coefficient matrix.

[0089] (3) After combination, the mask of each target instance is obtained, and the crop operation is performed on the mask, that is, the positions outside the instance frame are set to zero. The real labeled bounding box is used for training, and the bounding box obtained by the target detection part is used for testing.

[0090] (4) Taking 0.5 as the threshold, the output mask is binarized, the value range of the mask is limited to [0, 1], and the segmentation image of each instance anchor frame is obtained.

[0091] In this embodiment, by deduplicating the score bounding boxes of multiple categories, calculating the intersection over union between the same detection boxes and obtaining the intersection over union matrix, combining the mask coefficients and the object mask to obtain the segmented multiple object objects and background objects, the accuracy of object segmentation is effectively improved. By retaining the detection box with the highest score and reasonably screening, the interference of redundant boxes on the segmentation result is reduced, the segmentation effect is optimized, the situation of artifacts generated by dynamic targets in the process of neural radiation field modeling and model training not converging is reduced, and the training efficiency of the neural radiation field model is improved.

[0092] In this embodiment, by performing instance segmentation on each frame of image in the data set, combining the non-maximum suppression algorithm to deduplicate the score detection boxes, the object objects and background objects can be better segmented, the accuracy of the image segmentation result is improved, the influence of redundant detection boxes is reduced, the stability and reliability of subsequent neural radiation field modeling are improved, and the training efficiency of the neural radiation field model is improved.

[0093] In some embodiments, the cross-frame association of the object objects in different frames based on semantic consistency and spatial consistency comprises:

[0094] For each object object in each frame of image, it is judged whether the semantic class of the object object in the current frame and the object object in the next frame is the same, if the same, it is judged the proximity of the object object in the current frame and the object object in the next frame in space, if not the same, the object object in the current frame and the object object in the next frame are different object objects;

[0095] When the proximity is greater than or equal to a preset threshold, the object object in the current frame and the object object in the next frame are the same object object;

[0096] When the proximity is less than the preset threshold, the object object in the current frame and the object object in the next frame are different object objects.

[0097] It is easy to understand that after each frame of image is decoupled into background and different object objects, each object in each frame of image needs to be associated with the labeled object mask, and these object masks are predicted by using an instance segmentation network. Since the predicted object mask has no temporal consistency between different frames, the cross-frame association of the object objects in different frames is performed based on semantic consistency and spatial consistency, and the specific process is as follows:

[0098] Object association is performed between the previous frame and the current real-time frame: 1) semantic consistency: the object in the current frame is predicted to be the same semantic class as the previous frame, and this semantic information is obtained by the instance segmentation network. Only objects with the same semantics are likely to be the same object in the frame image. 2) spatial consistency: the object in the current frame is close to the object in the previous frame in space, which is measured by the intersection over union of their object boundaries.

[0099] For example, A represents the regression box of the object in the current frame, B represents the regression box of the object in other frames, A∩B is the intersection of the two regression boxes, and A∪B is the union of the two regression boxes. When the intersection-union ratio is greater than a threshold value 0.7, it is considered that the spatial positions of the two objects are consistent.

[0100] When the two criteria are met, it is considered that they are the same object instance and are represented by the same object model. Otherwise, they are different object instances, and for a new object instance in the system, a new object model is initialized and attached to the model stack. The objects in each frame of image are decoupled at the object level by semantic consistency and spatial consistency, and the objects between different frames are associated, and then the same objects between different frames are sent into the same local multi-layer perception network model for modeling.

[0101] In this embodiment, by comprehensively judging the semantic class and spatial proximity of the object in the current frame and the object in the next frame, the association of the same object between different frames is realized, the association error is reduced, the stability and accuracy of subsequent neural radiation field modeling are improved, and the generation quality of the neural radiation field model is improved.

[0102] In some embodiments, the hash coding of the background neural radiation field model and the plurality of object neural radiation field models based on linear discriminant analysis comprises:

[0103] Based on the background neural radiation field model and the plurality of object neural radiation field models, a target function is constructed respectively to obtain a background target function and a plurality of object target functions;

[0104] Based on the background target function and the plurality of object target functions, the background neural radiation field model and the plurality of object neural radiation field models are hash coded.

[0105] In some embodiments, the expression of the target function is as follows:

[0106] y(x)=sign(H T x)

[0107] Wherein, y(x) is a hash code, x is source data, and H is a mapping matrix.

[0108] Exemplarily, Figure 5 is a flowchart of hash coding based on linear discriminant analysis provided by the embodiments of the present application, as Figure 5As shown, the sampling point data is respectively encoded by direction encoding based on SH (Spherical Harmonics, spherical harmonics) spherical spectrum and position encoding based on multi-resolution LDAHE (Linear Discriminant Analysis, linear discriminant analysis), the position encoding is fused with the direction encoding through a 2-layer MLP (Multi-Layer Perceptron, multi-layer perceptron) network, and the flow of the hash encoding based on the linear discriminant analysis is as follows:

[0109] Given training data X = [x1, x2,..., x n ] ∈ R m×n , first, a data-aware objective function is constructed through local preservation and discriminative power enhancement, and then a hash function y(x) = sign(H T x) is generated by optimizing the objective function, where H ∈ R m ×d , x ∈ R m . Then, each data x i is mapped to a hash code y i through the hash function y(x), where similar data is mapped to similar hash codes, and different data is mapped to hash codes with large differences.

[0110] In the hash encoding process, the similarity relationship between the original data should be maintained as much as possible, and similar data should be mapped to similar hash codes to enable effective similarity retrieval in the hash space. Therefore, the hash encoding needs to construct a neighborhood graph to describe the local structure of the data. In the neighborhood graph, each point independently corresponds to a point in X. When a point is the k-neighbor of another point, the two points are connected by an edge, which is represented by a neighborhood matrix M, and the expression of the neighborhood matrix is as follows:

[0111]

[0112] Where M(i,j) is the weight edge between point x i and point x j , N k (x) represents the k-neighborhood set of the feature vector x.

[0113] The local preservation of the mapping function is evaluated by the preservation degree after mapping to the Hamming space through the neighborhood relationship, and the local preservation is quantified as semantic loss Loss, and the calculation formula is as follows:

[0114]

[0115] Where L is the Laplacian matrix, L = D - M = I - M, I is the identity matrix, and D(i,j) = ∑ jM(i, j) = 1.

[0116] Since the constraint of Y is a discrete requirement, directly solving the semantic loss is a non-deterministic polynomial problem. Therefore, Y is replaced by H T X, where X is the centralized data, i.e. The calculation formula of minimizing semantic loss is as follows:

[0117] min Tr(H T XLX T H)

[0118] s.t. H T H = I d

[0119] where I d is a dxd unit matrix, H T H = I d is a constraint condition to reduce inappropriate solutions.

[0120] where represents the kth class of centralized data, and X represents the centralized X. In order to be simple, the centralized original data is directly represented by X, and the expression of H is as follows:

[0121]

[0122] s.t. H = I d

[0123] where I m is an mxm unit matrix, and H T H = I d , the above formula is converted to:

[0124]

[0125] s.t. H T H = I d

[0126] Then the above formula is combined to form the overall objective function, and the calculation formula is as follows:

[0127]

[0128] s.t. H T H = I d

[0129] where 0≤λ≤1 is used to adjust the parameter, is a regularization term, and μ is a hyperparameter.

[0130] Let O = XLXT P = (XX T + μI m ) -1 XMX T , the objective function can be transformed into:

[0131]

[0132] s.t.H T H = I d

[0133] wherein the calculation formula of Q is as follows:

[0134] Q = O + λ(I m -P) + (1-λ)I m

[0135] = O + λI m -λP + I m -λI m

[0136] = O - λP + I m

[0137] The mapping matrix can be obtained by the eigenvectors corresponding to the d smallest non-zero eigenvalues of the matrix Q. On the basis of the mapping matrix, the background neural radiance field model and the plurality of object neural radiance field models are subjected to hash coding.

[0138] In this embodiment, by constructing the objective function and using the mapping matrix to hash code the source data, the source data can be effectively mapped into hash codes. This process improves the accuracy of feature representation of the background and the object in the neural radiance field model, thereby improving the effect of object recognition and classification. Through this method, the background neural radiance field model and the object neural radiance field model can be hash mapped in a more accurate coding space, optimizing the data storage and computing efficiency.

[0139] In this embodiment, by constructing the objective functions of the background neural radiance field model and the plurality of object neural radiance field models respectively, and performing hash coding based on the objective functions, the distinguishability of the background and the object can be effectively improved. By using linear discriminant analysis for hash coding optimization of the original neural radiance field position coding, the characteristics of the data are fully considered in the coding process, and by extracting key features and mapping them into hash codes, the hash codes can more accurately reflect the characteristics of the original data, effectively reducing the hash collision problem in the neural radiance field modeling process, and improving the modeling efficiency.

[0140] The embodiment of the application provides a linear discriminant hash coding modeling method based on front-back separation, and an execution subject can be a linear discriminant hash coding modeling device based on front-back separation. The embodiment of the application takes the linear discriminant hash coding modeling device based on front-back separation as an example to execute the linear discriminant hash coding modeling method based on front-back separation, and describes the linear discriminant hash coding modeling device based on front-back separation provided by the embodiment of the application.

[0141] The embodiment of the application also provides a linear discriminant hash coding modeling device based on front-back separation, as shown in the figure, the linear discriminant hash coding modeling device based on front-back separation comprises an acquisition module 610, a segmentation module 620, a marking module 630, an association module 640, an encoding module 650 and a modeling module 660. Figure 6

[0142] The acquisition module 610 is used for acquiring a data set of a target scene, and the data set comprises multiple frames of image data.

[0143] The segmentation module 620 is used for segmenting each frame of image in the data set based on an instance segmentation network to obtain multiple segmented object objects and background objects, and a background neural radiance field model is established based on the background objects.

[0144] The marking module 630 is used for marking each object object in each frame of image to obtain multiple object masks, and each object object corresponds to an object mask.

[0145] The association module 640 is used for cross-frame association of object objects in different frames based on semantic consistency and spatial consistency, and the same object object in different frames is input into the same local multi-layer perception network to establish a neural radiance field model, so that multiple object object neural radiance field models are obtained, and each object object corresponds to a neural radiance field model.

[0146] The encoding module 650 is used for hash coding of the background neural radiance field model and the multiple object object neural radiance field models based on linear discriminant analysis, so as to obtain a hash code corresponding to the background and a hash code corresponding to each object object.

[0147] The modeling module 660 is used for modeling based on the hash code corresponding to the background and the hash code corresponding to each object object, so as to obtain a target neural radiance field model, and the target neural radiance field model comprises the background and all object objects.

[0148] ​According to the linear discriminant hash coding modeling method based on front-back separation provided in the embodiment of the present application, by performing instance segmentation on each frame of image in the target scene data set, combining the background neural radiance field model and the plurality of object neural radiance field models, and using linear discriminant analysis for hash coding, the calculation complexity is effectively reduced, and the accuracy and efficiency of 3D scene reconstruction and object recognition are improved. By segmenting the background and foreground target objects and associating between different frames, respectively performing neural radiance field modeling, the artifacts caused by dynamic targets and the non-convergence of model training in the process of neural radiance field modeling can be reduced. Using linear discriminant hash coding effectively reduces the hash collision problem in the process of hash coding modeling, and improves the stability and reliability of neural radiance field modeling.

[0149] The linear discriminant hash coding modeling device based on front-back separation provided in the embodiment of the present application can realize each process of the linear discriminant hash coding modeling method embodiment based on front-back separation. Figures 1 to 5 The linear discriminant hash coding modeling device based on front-back separation provided in the embodiment of the present application can realize each process of the linear discriminant hash coding modeling method embodiment based on front-back separation.

[0150] In some embodiments, as shown in Figure 7 The embodiment of the present application also provides an electronic device 700, which includes a processor 701, a memory 702, and a computer program stored in the memory 702 and capable of running on the processor 701. When the processor 701 executes the program, each process of the above-mentioned linear discriminant hash coding modeling method embodiment based on front-back separation is realized, and the same technical effects can be achieved. To avoid repetition, details are not repeated here.

[0151] It should be noted that the electronic device in the embodiment of the present application includes the mobile electronic device and the non-mobile electronic device described above.

[0152] The embodiment of the present application also provides a non-transitory computer readable storage medium, which stores a computer program. When the processor executes the computer program, each process of the above-mentioned linear discriminant hash coding modeling method embodiment based on front-back separation is realized, and the same technical effects can be achieved. To avoid repetition, details are not repeated here.

[0153] The processor is the processor in the electronic device in the above-mentioned embodiments. The readable storage medium includes a computer readable storage medium, such as a computer readable memory (Read-Only Memory, ROM), a random access memory (Random Access Memory, RAM), a magnetic disk or an optical disk, etc.

[0154] The embodiment of the present application also provides a computer program product, which includes a computer program. When the processor executes the computer program, the above-mentioned linear discriminant hash coding modeling method based on front-back separation is realized.

[0155] The processor is a processor in the electronic device in the above embodiments. The readable storage medium includes a computer readable storage medium, such as a computer readable only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0156] The chip provided in the embodiments of the present application includes a processor and a communication interface, the communication interface is coupled with the processor, the processor is used to run programs or instructions to realize the processes of the above-mentioned linear discriminant hash coding modeling method based on front-back separation, and the same technical effects can be achieved. To avoid repetition, it will not be described here.

[0157] It should be understood that the chip mentioned in the embodiments of the present application can also be referred to as a device-level chip, a device chip, a chip device, or a system-on-chip device, etc.

[0158] It should be noted that in this paper, the terms "include", "contain" or any other variants thereof are intended to cover non-exclusive inclusion, so that the process, method, article or device including a series of elements not only includes those elements, but also includes other elements not explicitly listed or inherent to such process, method, article or device. Without more limitations, the element defined by the statement "including a…" does not exclude the presence of other identical elements in the process, method, article or device including the element. In addition, it should be pointed out that the scope of the methods and devices in the embodiments of the present application is not limited to performing functions in the order shown or discussed, but can also include performing functions in a substantially simultaneous manner or in reverse order according to the functions involved, for example, the described method can be performed in an order different from the described order, and various steps can also be added, omitted or combined. In addition, the features described with reference to certain examples can be combined in other examples.

[0159] From the above description of the embodiments, those skilled in the art can clearly understand that the above-mentioned embodiment method can be realized by means of software and necessary general hardware platform, of course, it can also be realized by hardware, but in many cases the former is a better embodiment. Based on such understanding, the technical solutions of the present application can be embodied in the form of a computer software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), including a plurality of instructions for making a terminal (which can be a mobile phone, computer, server, or network equipment, etc.) execute the linear discriminant hash coding modeling method based on front-back separation of the embodiments of the present application.

[0160] In the description of the present application, "first feature" and "second feature" can include one or more features.

[0161] In the description of the present application, the meaning of "a plurality" is two or more.

[0162] The embodiments of the present application are described above in connection with the accompanying drawings, but the present application is not limited to the above-described specific embodiments, and the above-described specific embodiments are merely illustrative, not restrictive, and a person of ordinary skill in the art can make many forms under the inspiration of the present application without departing from the purpose of the present application and the scope protected by the claims, all of which belong to the protection of the present application.

[0163] In the description of the present application, the description of the terms "one embodiment", "some embodiments", "an illustrative embodiment", "example", "specific example", or "some examples" means that the specific features, structures, materials or characteristics described in connection with the embodiment or example are included in at least one embodiment or example of the present application. In the present application, the illustrative description of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any one or more embodiments or examples in a suitable manner.

[0164] Although the embodiments of the present application have been shown and described, those of ordinary skill in the art can understand that various changes, modifications, replacements and variations can be made to the embodiments without departing from the principles and purposes of the present application, and the scope of the present application is defined by the claims and their equivalents.

Claims

1. A modeling method based on front-back separation linear discriminative hashing coding, characterized in that, The method comprises: acquiring a data set of a target scene, the data set comprising multiple frames of image data; segmenting each frame of image in the data set based on an instance segmentation network to obtain multiple segmented object objects and background objects, and establishing a background neural radiance field model based on the background objects; labeling each object object in each frame of image to obtain multiple object masks, each object object corresponding to an object mask; cross-frame associating the object objects in different frames based on semantic consistency and spatial consistency, inputting the same object objects in different frames into the same local multi-layer perception network to establish a neural radiance field model, and obtaining multiple object object neural radiance field models, each object object corresponding to a neural radiance field model; hash coding the background neural radiance field model and the multiple object object neural radiance field models based on linear discriminant analysis to obtain a hash code corresponding to the background and a hash code corresponding to each object object; modeling based on the hash code corresponding to the background and the hash code corresponding to each object object to obtain a target neural radiance field model, the target neural radiance field model comprising the background and all the object objects. 2.The front-back separation based linear discriminative hashing coding modeling method according to claim 1, wherein, The method comprises: inputting each frame of image in the data set into the instance segmentation network to obtain multiple category score detection boxes; de-duplicating the multiple category score detection boxes based on a non-maximum suppression algorithm to obtain the multiple segmented object objects and background objects. 3.The front-back separation based linear discriminative hashing coding modeling method according to claim 2, characterized in that, The method comprises: sorting the first n score detection boxes of each category according to scores, calculating the intersection over union between the detection boxes of the same category, and obtaining an intersection over union matrix based on the intersection over union between the detection boxes of the same category; based on the intersection over union matrix of each category, retaining the score detection box with the highest score result, and combining the mask coefficient and the object mask to obtain the multiple segmented object objects and background objects. 4.The front-back separation based linear discriminative hashing coding modeling method of claim 1, wherein, The method comprises: for each object object in each frame of image, determining whether the semantic category of the object object in the current frame is the same as that of the object object in the next frame, if yes, determining the spatial proximity of the object object in the current frame to the object object in the next frame, if not, the object object in the current frame and the object object in the next frame are different object objects; when the proximity is greater than or equal to a preset threshold, the object object in the current frame and the object object in the next frame are the same object object; when the proximity is less than the preset threshold, the object object in the current frame and the object object in the next frame are different object objects. 5.The front-back separation based linear discriminative hashing coding modeling method according to claim 1, wherein, The method comprises: constructing a target function based on the background neural radiance field model and the multiple object object neural radiance field models to obtain a background target function and multiple object object target functions; The background neural radiance field model and the plurality of object neural radiance field models are hash coded based on a background target function and a plurality of object target functions. 6.The front-back separation based linear discriminative hashing coding modeling method according to claim 5, characterized in that, An expression of the target function is as follows: y(x) = sign(H T x) wherein y(x) is a hash code, x is source data, and H is a mapping matrix. 7.The front-back separation based linear discriminative hashing coding modeling method according to claim 1, wherein, The instance segmentation network includes a backbone network, a prediction network, a prototype network, and a mask network, and the backbone network includes a feature pyramid network and a fully convolutional network.

8. A front-back separation based linear discriminative hashing coding modeling device, which is implemented by using the front-back separation based linear discriminative hashing coding modeling method in any one of claims 1 to 7, characterized in that, The device includes: An acquisition module configured to acquire a data set of a target scene, the data set including a plurality of image frames; A segmentation module configured to segment each image frame in the data set based on an instance segmentation network to obtain a plurality of segmented object and background objects, and to establish a background neural radiance field model based on the background objects; A labeling module configured to label each object in each image frame to obtain a plurality of object masks, each object corresponding to an object mask; An association module configured to perform cross-frame association on the objects in different frames based on semantic consistency and spatial consistency, to input the same object in different frames into the same local multi-layer perception network to establish a neural radiance field model, and to obtain a plurality of object neural radiance field models, each object corresponding to a neural radiance field model; A coding module configured to hash code the background neural radiance field model and the plurality of object neural radiance field models based on linear discriminant analysis to obtain a hash code corresponding to the background and a hash code corresponding to each object; A modeling module configured to model based on the hash code corresponding to the background and the hash code corresponding to each object to obtain a target neural radiance field model, the target neural radiance field model including the background and all the objects.

9. An electronic device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, The processor implements the front-back separation-based linear discriminant hash coding modeling method of any one of claims 1 to 7 when executing the program. 10.A non-transitory computer-readable storage medium having stored thereon a computer program, characterized in that, The computer program implements the front-back separation-based linear discriminant hash coding modeling method of any one of claims 1 to 7 when executed by the processor.

Images