A method for automatically capturing and acquiring material weighing images

Abstract

The present invention relates to the field of image communication technology, and in particular to an automatic capture method based on material weighing image screens. The method comprises the following steps: collecting information on on-site materials and vehicles based on multiple sensors, and performing point cloud registration processing based on a spatiotemporal coordinate system, thereby obtaining heterogeneous material voxel data, wherein the multiple sensors include structured light vision sensors, lidar sensors, and high-resolution camera sensors; performing visual context analysis on the heterogeneous material voxel data, thereby obtaining a visual context feature map; performing depth estimation on the visual context feature map based on structured light vision, and performing voxel-to-voxel ray tracing, thereby obtaining a three-dimensional geometric mesh model; segmenting the three-dimensional geometric mesh model into a number of tiny geometric facets, and obtaining micro-facet segmentation data. The present invention improves the adaptability to complex scenes by constructing a visual context feature map to comprehensively perceive objects, vehicles, and scene contexts.

Description

Technical Field

[0001] The present invention relates to the field of image communication technology, and in particular to an automatic capture and acquisition method based on material weighing image screens. Background Art

[0002] Material weighing image screen refers to the image screen captured in real time by the camera equipment during the material weighing process. This image screen usually includes the entire scene of the material placed on the weighing equipment, which is used to record the process and status of material weighing. The automatic capture acquisition method refers to the real-time capture of images or video screens by an automated system in a specific scene in order to record and analyze important events or processes. The automatic capture system of material weighing image screen is usually integrated into the material weighing system, including components such as floor scales, weighing sensors, video surveillance cameras, image processing software and databases.

[0003] The automatic capture and acquisition method of material weighing images plays an important role in many practical application scenarios such as logistics distribution, enterprise material management, unattended weighing systems and intelligent weighbridge systems. However, the traditional automatic capture and acquisition method based on material weighing images often has some problems: to achieve accurate weighing in the weighbridge system, it is necessary to accurately locate the position of the vehicle and materials on the weighbridge, which has high requirements on the camera position, viewing angle, resolution, etc. If the positioning is inaccurate, it will affect the weighing accuracy; due to the possible existence of occlusion, lighting changes, viewing angle changes and other factors in the real scene, it brings difficulties to the accurate detection and identification of objects, such as identifying the type of bulk goods in the carriage, or segmenting individual items in a cluttered scene. Summary of the invention

[0004] Based on this, it is necessary for the present invention to provide a method for automatically capturing and acquiring material weighing images to solve at least one of the above technical problems.

[0005] To achieve the above purpose, a method for automatically capturing and acquiring material weighing images includes the following steps:

[0006] Step S1: collecting information on on-site materials and vehicles based on multiple sensors, and performing point cloud registration processing based on a spatiotemporal coordinate system, thereby obtaining voxel data of heterogeneous materials, wherein the multiple sensors include structured light vision sensors, lidar sensors, and high-resolution camera sensors;

[0007] Step S2: Perform visual context analysis on the heterogeneous material voxel data to obtain a visual context feature map; perform depth estimation based on structured light vision on the visual context feature map, and perform voxel-to-voxel ray tracing to obtain a three-dimensional geometric mesh model;

[0008] Step S3: Segment the three-dimensional geometric grid model into a number of tiny geometric facets to obtain micro-facet segmentation data; perform optical property analysis on the micro-facet segmentation data according to the heterogeneous material voxel data, and perform data fusion with the three-dimensional geometric grid model to obtain a polarized three-dimensional grid model;

[0009] Step S4: using a variational autoencoder to perform visual decoupling of the intrinsic representation of the polarized three-dimensional grid model, thereby obtaining an intrinsic semantic feature tensor; projecting the intrinsic semantic feature tensor into a molecular orbital-like subspace, and performing three-dimensional reconstruction based on instance segmentation and occlusion region reasoning, thereby obtaining a molecular orbital-like topological map;

[0010] Step S5: accurately locate and identify vehicles and materials based on the molecular orbital topology map and preset targets, thereby obtaining re-identification metadata.

[0011] The present invention can obtain voxel data of heterogeneous materials through multi-sensor information collection and point cloud registration processing. These data can provide detailed three-dimensional information of on-site materials and vehicles, including shape, position and spatial coordinates. The multi-sensor includes structured light vision sensor, lidar sensor and high-resolution camera sensor, and their combination can provide rich perception capabilities. Visual context analysis of heterogeneous material voxel data can obtain visual context feature map, which can provide more advanced semantic information, including object category, structure and relationship. Depth estimation and voxel-to-voxel ray tracing through structured light vision can obtain a three-dimensional geometric grid model. This model can more accurately represent the shape and surface geometry of the object. The three-dimensional geometric grid model is divided into tiny geometric facets, and optical property analysis and data fusion are performed to obtain a polarized three-dimensional grid model. This model can provide surface properties and material information of the object, and combined with the geometric structure, it provides a more realistic and detailed representation. By using a variational autoencoder to perform visual decoupling of the inherent representation of the polarized three-dimensional grid model, an inherent semantic feature tensor can be obtained; this feature tensor can capture the inherent semantic features of the object and provide a more advanced object representation and semantic understanding. The inherent semantic feature tensor is projected into a molecular orbital subspace, and a three-dimensional reconstruction based on instance segmentation and occlusion region reasoning is performed to obtain a molecular orbital topology map, which can provide the relationship and connection information between objects and help understand the organizational structure and layout of objects in the scene. According to the molecular orbital topology map and the preset target, the vehicle and materials are accurately positioned and identified, and re-identification metadata can be obtained; these metadata may include the identification, attributes and location information of the vehicle and materials, and provide accurate positioning and identification results for the target. Through the processing and analysis of the above steps, rich object information can be extracted from multi-sensor data, including shape, position, surface attributes, inherent semantic features and relationships, etc.; this information can be used for applications such as scene understanding, target recognition, three-dimensional reconstruction and precise positioning. The present invention provides more accurate and detailed object representation, enhances scene understanding and decision-making capabilities, and supports applications in the fields of intelligent transportation, logistics management and security monitoring. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] Other features, objects and advantages of the present invention will become more apparent from the detailed description of non-limiting embodiments thereof made with reference to the following drawings:

[0013] Figure 1 It is a schematic diagram of the steps of the automatic capture and acquisition method based on the material weighing image screen of the present invention;

[0014] Figure 2 for Figure 1 Detailed step flow diagram of step S1;

[0015] Figure 3 for Figure 1 Detailed step flow chart of step S2 in FIG. DETAILED DESCRIPTION

[0016] The technical method of the present invention is described clearly and completely below in conjunction with the accompanying drawings. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by technicians in this field without creative work are within the scope of protection of the present invention.

[0017] In addition, the accompanying drawings are only schematic illustrations of the present invention and are not necessarily drawn to scale. The same reference numerals in the figures represent the same or similar parts, and their repeated description will be omitted. Some of the block diagrams shown in the accompanying drawings are functional entities and do not necessarily correspond to physically or logically independent entities. The functional entities can be implemented in software form, or implemented in one or more hardware modules or integrated circuits, or implemented in different networks and / or processor methods and / or microcontroller methods.

[0018] It should be understood that, although the terms "first", "second", etc. may be used herein to describe various units, these units should not be limited by these terms. These terms are used only to distinguish one unit from another unit. For example, without departing from the scope of the exemplary embodiments, the first unit may be referred to as the second unit, and similarly the second unit may be referred to as the first unit. The term "and / or" used herein includes any and all combinations of one or more of the listed associated items.

[0019] To achieve this, please refer to Figures 1 to 3 The present invention provides a method for automatically capturing and acquiring material weighing images, the method comprising the following steps:

[0020] Step S1: collecting information on on-site materials and vehicles based on multiple sensors, and performing point cloud registration processing based on a spatiotemporal coordinate system, thereby obtaining voxel data of heterogeneous materials, wherein the multiple sensors include structured light vision sensors, lidar sensors, and high-resolution camera sensors;

[0021] Step S2: Perform visual context analysis on the heterogeneous material voxel data to obtain a visual context feature map; perform depth estimation based on structured light vision on the visual context feature map, and perform voxel-to-voxel ray tracing to obtain a three-dimensional geometric mesh model;

[0022] Step S3: Segment the three-dimensional geometric grid model into a number of tiny geometric facets to obtain micro-facet segmentation data; perform optical property analysis on the micro-facet segmentation data according to the heterogeneous material voxel data, and perform data fusion with the three-dimensional geometric grid model to obtain a polarized three-dimensional grid model;

[0023] Step S4: using a variational autoencoder to perform visual decoupling of the intrinsic representation of the polarized three-dimensional grid model, thereby obtaining an intrinsic semantic feature tensor; projecting the intrinsic semantic feature tensor into a molecular orbital-like subspace, and performing three-dimensional reconstruction based on instance segmentation and occlusion region reasoning, thereby obtaining a molecular orbital-like topological map;

[0024] Step S5: accurately locate and identify vehicles and materials based on the molecular orbital topology map and preset targets, thereby obtaining re-identification metadata.

[0025] In the embodiment of the present invention, reference Figure 1 The above is a schematic flow chart of the steps of an automatic capture method based on a material weighing image screen of the present invention. In this example, the automatic capture method based on a material weighing image screen includes the following steps:

[0026] Step S1: collecting information on on-site materials and vehicles based on multiple sensors, and performing point cloud registration processing based on a spatiotemporal coordinate system, thereby obtaining voxel data of heterogeneous materials, wherein the multiple sensors include structured light vision sensors, lidar sensors, and high-resolution camera sensors;

[0027] The embodiment of the present invention uses structured light vision sensors, lidar sensors and high-resolution camera sensors to simultaneously collect information on on-site materials and vehicles; the collected data is processed by point cloud registration in a spatiotemporal coordinate system to ensure that data collected by different sensors can be aligned in the same coordinate system; based on the registered point cloud data, it is converted into voxel data to form heterogeneous material voxel data.

[0028] Step S2: Perform visual context analysis on the heterogeneous material voxel data to obtain a visual context feature map; perform depth estimation based on structured light vision on the visual context feature map, and perform voxel-to-voxel ray tracing to obtain a three-dimensional geometric mesh model;

[0029] The embodiments of the present invention perform visual context analysis on heterogeneous material voxel data, extract visual features such as color, texture, shape, etc. through image processing and computer vision algorithms; perform depth estimation based on structured light vision, and use structured light projection and reflection models to infer the depth information of the object surface; perform voxel-to-voxel ray tracing, and obtain a three-dimensional geometric mesh model by emitting rays from the camera position and calculating the intersection with the voxels.

[0030] Step S3: Segment the three-dimensional geometric grid model into a number of tiny geometric facets to obtain micro-facet segmentation data; perform optical property analysis on the micro-facet segmentation data according to the heterogeneous material voxel data, and perform data fusion with the three-dimensional geometric grid model to obtain a polarized three-dimensional grid model;

[0031] The embodiment of the present invention segments a three-dimensional geometric grid model into tiny geometric face elements to form micro-facet segmentation data; performs optical property analysis on the micro-facet segmentation data, and assigns corresponding optical properties to each micro-facet by analyzing the optical information in the voxel data of heterogeneous materials, such as reflectivity, transparency, etc.; performs data fusion of the optical properties with the three-dimensional geometric grid model, and associates the optical properties with each geometric face element to obtain a polarized three-dimensional grid model.

[0032] Step S4: using a variational autoencoder to perform visual decoupling of the intrinsic representation of the polarized three-dimensional grid model, thereby obtaining an intrinsic semantic feature tensor; projecting the intrinsic semantic feature tensor into a molecular orbital-like subspace, and performing three-dimensional reconstruction based on instance segmentation and occlusion region reasoning, thereby obtaining a molecular orbital-like topological map;

[0033] The embodiment of the present invention uses a variational autoencoder to train a polarized three-dimensional grid model, learns its inherent representation, and thus obtains an inherent semantic feature tensor; the inherent semantic feature tensor is projected into a molecular orbital subspace, and a representation of a molecular orbital topology map is obtained by mapping the feature tensor into the molecular orbital feature space; based on the method of instance segmentation and occlusion region reasoning, the molecular orbital topology map is reconstructed in three dimensions. This includes using an instance segmentation algorithm to segment different object instances and inferring possible occlusion regions, thereby obtaining a more complete and accurate molecular orbital topology map.

[0034] Step S5: accurately locate and identify vehicles and materials based on the molecular orbital topology map and preset targets, thereby obtaining re-identification metadata.

[0035] The embodiment of the present invention utilizes the molecular orbital-like topological map and the information of preset targets, and uses the target recognition algorithm to accurately locate and identify vehicles and materials; through the detection and tracking algorithm, the target information in the molecular orbital-like topological map is used to achieve the positioning and tracking of the target, thereby obtaining the re-identification metadata, including the recognition result, location information and trajectory of the target.

[0036] The present invention can obtain voxel data of heterogeneous materials through multi-sensor information collection and point cloud registration processing. These data can provide detailed three-dimensional information of on-site materials and vehicles, including shape, position and spatial coordinates. The multi-sensor includes structured light vision sensor, lidar sensor and high-resolution camera sensor, and their combination can provide rich perception capabilities. Visual context analysis of heterogeneous material voxel data can obtain visual context feature map, which can provide more advanced semantic information, including object category, structure and relationship. Depth estimation and voxel-to-voxel ray tracing through structured light vision can obtain a three-dimensional geometric grid model. This model can more accurately represent the shape and surface geometry of the object. The three-dimensional geometric grid model is divided into tiny geometric facets, and optical property analysis and data fusion are performed to obtain a polarized three-dimensional grid model. This model can provide surface properties and material information of the object, and combined with the geometric structure, it provides a more realistic and detailed representation. By using a variational autoencoder to perform visual decoupling of the inherent representation of the polarized three-dimensional grid model, an inherent semantic feature tensor can be obtained; this feature tensor can capture the inherent semantic features of the object and provide a more advanced object representation and semantic understanding. The inherent semantic feature tensor is projected into a molecular orbital subspace, and a three-dimensional reconstruction based on instance segmentation and occlusion region reasoning is performed to obtain a molecular orbital topology map, which can provide the relationship and connection information between objects and help understand the organizational structure and layout of objects in the scene. According to the molecular orbital topology map and the preset target, the vehicle and materials are accurately positioned and identified, and re-identification metadata can be obtained; these metadata may include the identification, attributes and location information of the vehicle and materials, and provide accurate positioning and identification results for the target. Through the processing and analysis of the above steps, rich object information can be extracted from multi-sensor data, including shape, position, surface attributes, inherent semantic features and relationships, etc.; this information can be used for applications such as scene understanding, target recognition, three-dimensional reconstruction and precise positioning. The present invention provides more accurate and detailed object representation, enhances scene understanding and decision-making capabilities, and supports applications in the fields of intelligent transportation, logistics management and security monitoring.

[0037] Preferably, step S1 comprises the following steps:

[0038] Step S11: projecting a coded light field onto on-site materials and vehicles through a structured light vision sensor, and capturing reflected coded light, thereby obtaining depth image data, wherein the structured light vision sensor is composed of a projector and a camera;

[0039] Step S12: using a laser radar sensor to emit laser beams to on-site materials and vehicles, measure the time difference of reflected light, and calculate the distance, thereby obtaining three-dimensional point cloud data;

[0040] Step S13: obtaining high-definition RGB image data of the scene through a high-resolution camera sensor;

[0041] Step S14: performing spatiotemporal registration processing on the depth image data and the three-dimensional point cloud data based on the spatiotemporal coordinate system, thereby generating structured light point cloud data;

[0042] Step S15: projecting the structured light point cloud data into a unified three-dimensional grid voxel, thereby obtaining three-dimensional grid voxel data, wherein the voxel includes a geometric position and a surface normal vector;

[0043] Step S16: statistically coloring each voxel of the three-dimensional grid voxel data according to the RGB image data, so as to obtain heterogeneous material voxel data.

[0044] As an embodiment of the present invention, refer to Figure 2 As shown, Figure 1 Detailed step flow diagram of step S1 in the embodiment of the present invention, step S1 includes the following steps:

[0045] Step S11: projecting a coded light field onto on-site materials and vehicles through a structured light vision sensor, and capturing reflected coded light, thereby obtaining depth image data, wherein the structured light vision sensor is composed of a projector and a camera;

[0046] The embodiment of the present invention uses a structured light vision sensor, which consists of a projector and a camera; the projector projects a specific coded light field onto on-site materials and vehicles; the camera captures the reflected coded light and records the light intensity of each pixel; by comparing the phase difference of the coded light during the projection and capture process, the depth information of each pixel is calculated; and the calculated depth information is converted into depth image data.

[0047] Step S12: using a laser radar sensor to emit laser beams to on-site materials and vehicles, measure the time difference of reflected light, and calculate the distance, thereby obtaining three-dimensional point cloud data;

[0048] An embodiment of the present invention uses a laser radar sensor, which emits a laser beam and receives reflected light; the laser radar sensor emits a short-pulse laser beam; the sensor receives the reflected light and measures the time difference between the emission and reception of the laser beam; the distance between the object and the sensor is calculated by measuring the time difference and the speed of light; and the calculated distance information is converted into three-dimensional point cloud data.

[0049] Step S13: obtaining high-definition RGB image data of the scene through a high-resolution camera sensor;

[0050] The embodiment of the present invention uses a high-resolution camera sensor that can capture high-definition RGB image data; the camera is placed at an appropriate position and angle to ensure that the required scene conditions can be captured; the camera captures RGB image data of objects and vehicles on the scene.

[0051] Step S14: performing spatiotemporal registration processing on the depth image data and the three-dimensional point cloud data based on the spatiotemporal coordinate system, thereby generating structured light point cloud data;

[0052] The embodiments of the present invention use positioning information or the built-in positioning function of the sensor to establish a spatiotemporal coordinate system to ensure that the depth image data and the three-dimensional point cloud data are aligned in the same coordinate system; the depth image data and the three-dimensional point cloud data are spatiotemporally registered so that they are aligned in the same coordinate system; the registered depth image data and the three-dimensional point cloud data are fused to generate structured light point cloud data.

[0053] Step S15: projecting the structured light point cloud data into a unified three-dimensional grid voxel, thereby obtaining three-dimensional grid voxel data, wherein the voxel includes a geometric position and a surface normal vector;

[0054] The embodiment of the present invention determines the size and resolution of the three-dimensional grid and divides the scene range into a series of voxels; for each voxel, the points in the structured light point cloud data are projected into the corresponding voxel; for each voxel, its geometric position is calculated, and the average position of the points in the voxel or other methods may be selected; for each voxel, its surface normal vector is calculated, and the average normal vector of the points in the voxel or other methods may be selected; the calculated geometric position and surface normal vector information are associated with the voxel to obtain three-dimensional grid voxel data.

[0055] Step S16: statistically coloring each voxel of the three-dimensional grid voxel data according to the RGB image data, so as to obtain heterogeneous material voxel data.

[0056] The embodiment of the present invention uses RGB image data: using the RGB image data obtained in step S13; for each voxel, statistical coloring is performed according to the color information of the points inside the voxel; statistics are performed according to the RGB color values ​​of the points inside the voxel, and a histogram or other statistical methods may be used; based on the statistical results, the color representation of each voxel is determined, and the mode, average or other methods may be used; the determined voxel color information is associated with the voxel to obtain heterogeneous material voxel data.

[0057] The present invention can obtain depth image data by projecting coded light fields and capturing reflected coded light of structured light vision sensors. The structured light vision sensor is composed of a projector and a camera. The projector projects the coded light field onto the scene, and the camera captures the reflected light and calculates the depth according to the coded information; in this way, the depth information of materials and vehicles in the scene can be obtained, thereby providing more accurate distance measurement. By using a laser radar sensor to emit a laser beam and measure the time difference of the reflected light, the distance of the materials and vehicles can be calculated; by scanning multiple laser beams, three-dimensional point cloud data can be obtained, which contains the spatial position and shape information of the object; this provides an accurate, non-contact measurement method suitable for complex scenes and different surface materials. High-definition RGB image data of the scene is obtained through a high-resolution camera sensor; these image data contain rich visual information of objects and scenes, and can be used for subsequent image processing and analysis. The depth image data and the three-dimensional point cloud data are subjected to spatiotemporal registration processing to align them in a spatiotemporal coordinate system; in this way, the data of different sensors can be fused, and the registration error can be reduced to generate structured light point cloud data. The structured light point cloud data integrates depth and three-dimensional information, providing a more comprehensive and accurate representation of objects. The structured light point cloud data is projected into a unified three-dimensional grid voxel to obtain three-dimensional grid voxel data. In this way, the point cloud data can be converted into a discrete voxel representation, each voxel includes a geometric position and a surface normal vector; the voxel representation provides a compact and easy-to-process representation for subsequent analysis and processing. The three-dimensional grid voxel data is colored according to the RGB image data. By statistically coloring each voxel, the color information can be associated with the geometric information to obtain heterogeneous material voxel data, so that color attributes can be added to the three-dimensional representation to provide a richer and more realistic object representation. Through the processing and analysis of the above steps, rich object information can be obtained, including depth, geometry, surface normal vectors and color, which can be used for applications such as object recognition, scene analysis, three-dimensional modeling and visualization. Preferably, it includes providing accurate depth measurement, comprehensive three-dimensional information, high-resolution visual data and realistic object representation, so as to support applications in the fields of intelligent transportation, security monitoring, virtual reality, etc.

[0058] Preferably, step S2 comprises the following steps:

[0059] Step S21: construct an encoder-decoder network based on 3D convolution, input the heterogeneous material voxel data into the encoder-decoder network, extract local and global context features layer by layer through the 3D convolution of the encoder, and obtain context encoding feature data;

[0060] Step S22: performing visual context analysis on the context encoding feature data through a decoder to obtain a visual context feature map, wherein the decoder includes a semantic segmentation branch, a depth of field estimation branch, and an illumination analysis branch;

[0061] Step S23: using the depth image data as a priori, fusing the context features and the depth priori into the visual context feature map, and performing depth value regression to obtain a high-precision depth voxel map;

[0062] Step S24: using the depth voxel map as the distance field to perform back-projection of each voxel in the three-dimensional space, and emitting sight tracking one by one from the camera perspective until hitting the surface of other voxels, calculating the three-dimensional coordinates and surface normal vector of the hitting point, thereby obtaining sight tracking result data;

[0063] Step S25: construct triangular facets according to adjacent normal vectors in the line of sight tracking result data, and perform facet refinement based on computer graphics smoothing technology to obtain a three-dimensional geometric mesh closed model without holes;

[0064] Step S26: Perform Laplace smoothing on the three-dimensional geometric mesh closed model, and perform detail enhancement according to the visual context feature map, so as to obtain a three-dimensional geometric mesh model.

[0065] As an embodiment of the present invention, refer to Figure 3 As shown, Figure 1 Detailed step flow diagram of step S2 in the embodiment of the present invention, step S2 includes the following steps:

[0066] Step S21: construct an encoder-decoder network based on 3D convolution, input the heterogeneous material voxel data into the encoder-decoder network, extract local and global context features layer by layer through the 3D convolution of the encoder, and obtain context encoding feature data;

[0067] The embodiment of the present invention uses a deep learning framework (such as TensorFlow or PyTorch) to build an encoder-decoder network based on 3D convolution; heterogeneous material voxel data is used as the input of the encoder-decoder network; local and global context features of the heterogeneous material voxel data are gradually extracted through multi-layer 3D convolution operations; the output of the encoder is context encoded feature data, which contains feature information extracted through the convolution operation.

[0068] Step S22: performing visual context analysis on the context encoding feature data through a decoder to obtain a visual context feature map, wherein the decoder includes a semantic segmentation branch, a depth of field estimation branch, and an illumination analysis branch;

[0069] The embodiment of the present invention adds a decoder branch to the encoder-decoder network, and the decoder branch analyzes the context encoding feature data, including semantic segmentation, depth of field estimation and illumination analysis; uses convolution and upsampling operations to perform semantic segmentation on the context encoding feature data to generate a pixel-level semantic label map; through convolution and upsampling operations, depth of field estimation is performed on the context encoding feature data to generate a depth map; through convolution and upsampling operations, illumination analysis is performed on the context encoding feature data to generate an illumination map; the visual context feature map includes a semantic label map, a depth of field map and an illumination map.

[0070] Step S23: using the depth image data as a priori, fusing the context features and the depth priori into the visual context feature map, and performing depth value regression to obtain a high-precision depth voxel map;

[0071] The depth image data obtained by the embodiment of the present invention is used as prior information, and the visual context feature map is fused with the depth image data. Convolution, upsampling or other fusion methods can be used; the fused features are processed, and the precise depth value of each voxel is predicted by a regression method; the result of the depth value regression is mapped to the voxel grid to obtain a high-precision depth voxel map.

[0072] Step S24: using the depth voxel map as the distance field to perform back-projection of each voxel in the three-dimensional space, and emitting sight tracking one by one from the camera perspective until hitting the surface of other voxels, calculating the three-dimensional coordinates and surface normal vector of the hitting point, thereby obtaining sight tracking result data;

[0073] The embodiment of the present invention regards the depth voxel map as a distance field, in which the depth value of each voxel represents the distance from the camera; starting from the camera perspective, the depth value of each voxel is back-projected into the three-dimensional space to obtain the position of the voxel in the world coordinate system; for each voxel, a ray is emitted along the line of sight direction from the camera perspective, and it is determined one by one whether the ray intersects with the surfaces of other voxels; for each ray, an intersection calculation is performed with the voxel grid to determine whether the ray hits the surfaces of other voxels; when the ray hits the surfaces of other voxels, the three-dimensional coordinates and surface normal vectors of the hitting point are calculated; the three-dimensional coordinates and surface normal vectors of all hitting points are used as line of sight tracking result data.

[0074] Step S25: construct triangular facets according to adjacent normal vectors in the line of sight tracking result data, and perform facet refinement based on computer graphics smoothing technology to obtain a three-dimensional geometric mesh closed model without holes;

[0075] The embodiment of the present invention uses the three-dimensional coordinates and surface normal vectors in the line of sight tracking result data to calculate the normal vector information between adjacent hit points; according to the adjacent normal vector information, the adjacent hit points are connected to construct triangular facets; the constructed triangular facets are refined and the smoothing technology of computer graphics (such as subdivision surface, smoothing filtering, etc.) is used to make the geometric mesh surface smoother and continuous; after the facet construction and refinement processing, a three-dimensional geometric mesh closed model without holes is obtained.

[0076] Step S26: Perform Laplace smoothing on the three-dimensional geometric mesh closed model, and perform detail enhancement according to the visual context feature map, so as to obtain a three-dimensional geometric mesh model.

[0077] The embodiment of the present invention performs Laplace smoothing on the three-dimensional geometric mesh closed model, and makes the surface of the mesh model smoother by adjusting the vertex positions and normal vectors; uses the information in the visual context feature map to perform detail enhancement operations on the three-dimensional geometric mesh model, such as texture mapping the model according to the semantic segmentation results; after Laplace smoothing and detail enhancement processing, the final three-dimensional geometric mesh model is obtained.

[0078] The present invention constructs an encoder-decoder network based on 3D convolution to process heterogeneous material voxel data. The encoder extracts local and global context features through layer-by-layer 3D convolution operations. These features represent the shape, structure and relationship of the object. The decoder can obtain context encoding feature data by decoding and reconstructing the encoding features, so that the ability of deep learning can be used to extract meaningful feature representations and provide input for subsequent visual context analysis. The decoder performs visual context analysis on the context encoding feature data. The decoder includes a semantic segmentation branch, a depth estimation branch and an illumination analysis branch. The semantic segmentation branch can classify each voxel into different semantic categories to obtain the semantic information of the object. The depth estimation branch can estimate the depth value of each voxel and provide the distance information of the object. The illumination analysis branch can analyze the illumination conditions and provide illumination information. In this way, different aspects of visual context can be comprehensively considered to provide rich features for the subsequent generation of deep voxel maps. Taking the depth image data as the prior, the context features and depth priors are integrated into the visual context feature map, and the depth value regression is performed. By combining the depth image data with the visual context feature map, the traditional depth prior information and the learning-based visual context features can be integrated; the depth value regression can further improve the accuracy of depth estimation, thereby obtaining a high-precision depth voxel map, which can obtain more accurate three-dimensional spatial information and provide more reliable depth data for subsequent line of sight tracking. Taking the depth voxel map as the distance field, each voxel is back-projected into three-dimensional space, and line of sight tracking is performed. By emitting the line of sight one by one from the camera perspective and tracking until it hits the surface of other voxels, the three-dimensional coordinates and surface normal vector of the hitting point can be calculated, so that the geometric shape and surface information of the objects in the scene can be obtained, providing a data basis for the subsequent three-dimensional geometric mesh generation. According to the adjacent normal vectors in the gaze tracking result data, triangular facets are constructed and facet refinement is performed. By connecting the normal vectors of adjacent hit points, triangular facets can be constructed. Facet refinement technology can further improve the quality and smoothness of the facets, remove possible holes and discontinuities, and thus obtain a hole-free three-dimensional geometric mesh closed model, which can generate a more realistic and detailed object surface representation. The three-dimensional geometric mesh closed model is Laplace smoothed and detailed enhancement is performed according to the visual context feature map. Laplace smoothing can further eliminate surface noise and irregularities and make the model smoother; at the same time, the model can be enhanced in detail according to the visual context feature map to make it richer and more realistic, so that a high-quality and detailed three-dimensional geometric mesh model can be obtained. Through the processing and analysis of the above steps, high-precision deep voxel maps and three-dimensional geometric mesh models can be obtained. These models have accurate depth information, real geometric shapes and details, and can be used for applications such as object recognition, scene reconstruction, and virtual reality.The present invention provides high-precision depth estimation, complete geometric representation, detailed models and more realistic visual perception, thereby supporting applications in the fields of intelligent transportation, augmented reality, 3D visualization and so on.

[0079] Preferably, step S22 includes the following steps:

[0080] Step S221: performing pixel-level semantic segmentation based on different entities on the context encoding feature data through the semantic segmentation branch, thereby obtaining a semantic segmentation voxel atlas;

[0081] The embodiments of the present invention can use a deep learning framework (such as TensorFlow, PyTorch) and a semantic segmentation model (such as U-Net, DeepLab); prepare a labeled training data set, including input context encoding feature data and corresponding pixel-level semantic label data; build a semantic segmentation model, use the context encoding feature data as input, extract features layer by layer through convolution and upsampling operations, and perform pixel-level classification; use the training data set to train the semantic segmentation model, adjust the model parameters by optimizing the loss function, so that it can accurately perform semantic segmentation on different entities; use the trained semantic segmentation model to infer the new context encoding feature data, obtain the semantic label of each voxel, and generate a semantic segmentation voxel map.

[0082] Step S222: using the depth estimation branch to estimate the depth value of each voxel of the context encoding feature data, thereby obtaining a high-precision depth voxel map;

[0083] The embodiment of the present invention uses a deep learning framework and a depth estimation model (such as Monodepth and DPSNet) to prepare a training data set with real depth values, including input context encoding feature data and corresponding real depth value data; constructs a depth estimation model, takes the context encoding feature data as input, and estimates the depth value of each voxel layer by layer through convolution and upsampling operations; uses the training data set to train the depth estimation model, and adjusts the model parameters by optimizing the loss function so that it can accurately estimate the depth value of each voxel; uses the trained depth estimation model to infer the new context encoding feature data to obtain the depth value of each voxel, and generates a high-precision depth voxel map.

[0084] Step S223: performing illumination distribution analysis of the shadow degree and reflection intensity of each voxel on the context encoding feature data through the illumination analysis branch, thereby obtaining an illumination voxel atlas;

[0085] The embodiment of the present invention uses a deep learning framework and an illumination parsing model (such as Neural Lighting) to prepare a training data set with real illumination information, including input context encoding feature data and corresponding real illumination information data; constructs an illumination parsing model, takes the context encoding feature data as input, and analyzes the shadow degree and reflection intensity of each voxel layer by layer through convolution and full connection operations; uses the training data set to train the illumination parsing model, and adjusts the model parameters by optimizing the loss function so that the model can accurately analyze the shadow degree and reflection intensity of each voxel; uses the trained illumination parsing model to infer the new context encoding feature data, obtains the shadow degree and reflection intensity of each voxel, and generates an illumination voxel map.

[0086] Step S224: accurately align the semantic segmentation voxel map, the high-precision depth of field voxel map, and the illumination voxel map in space and semantics, and fuse them into a visual context feature map.

[0087] The embodiment of the present invention uses an image processing library (such as OpenCV) and an image registration algorithm (such as ICP, image pyramid) to spatially align the semantic segmentation voxel map, the high-precision depth of field voxel map and the illumination voxel map to ensure that they are in the same coordinate space; according to the semantic label information in the semantic segmentation voxel map, the high-precision depth of field voxel map and the illumination voxel map are semantically aligned, that is, voxels with the same semantics are aligned together; the aligned semantic segmentation voxel map, the high-precision depth of field voxel map and the illumination voxel map are fused, and weighted fusion or pixel-by-pixel operation can be used to obtain a visual context feature map; the generated visual context feature map can be used for further computer vision tasks, such as target detection, scene understanding, etc.

[0088] The present invention performs pixel-level semantic segmentation on context-encoded feature data through a semantic segmentation branch, and can classify each voxel into different semantic entities, thereby obtaining a semantic segmentation voxel map. This map provides semantic understanding and distinction of different objects in the scene, and provides accurate semantic information for subsequent analysis and processing. It can be used for tasks such as object recognition, scene understanding, and intelligent decision-making, which helps to improve the system's ability to understand the environment. The depth of field estimation branch is used to estimate the depth of field value of the context-encoded feature data, and can provide accurate depth of field information for each voxel, obtaining a high-precision depth of field voxel map. This map can reflect the distance information of objects in the scene, which is very important for realizing applications such as virtual-real fusion, depth perception, and stereoscopic vision. It can be used in fields such as visual ranging, three-dimensional reconstruction, and augmented reality, providing more realistic and accurate depth perception. By performing illumination analysis on the context-encoded feature data through the illumination parsing branch, the shadow degree and reflection intensity of each voxel can be estimated, thereby obtaining an illumination voxel map. This map can provide the distribution and changes of illumination in the scene, and is important for achieving realistic rendering, illumination reconstruction, and light and shadow effects. It can be used in computer graphics, game development, and virtual reality to provide more realistic and true lighting effects. The semantic segmentation voxel map, high-precision depth of field voxel map, and illumination voxel map are spatially and semantically aligned, and fused into a visual context feature map. This map combines semantic information, depth information, and illumination information, and provides rich visual context features. It can be used for tasks such as scene understanding, image segmentation, 3D reconstruction, and virtual reality, helping the system to better understand and perceive the environment and support more advanced visual analysis and decision-making.

[0089] Preferably, step S3 comprises the following steps:

[0090] Step S31: Segment the three-dimensional geometric mesh model into a number of tiny triangular facets, each of which corresponds to a tiny area on the surface of the object, thereby obtaining microfacet segmentation data, wherein the microfacet segmentation data includes geometric information of vertices and normal vectors of each microfacet;

[0091] The embodiment of the present invention uses a computer graphics library (such as OpenGL, Three.js) to process and segment the geometric mesh model, prepares data of the three-dimensional geometric mesh model, including vertex coordinates and face information; divides the three-dimensional geometric mesh model into a number of tiny triangular face elements, which can be achieved using a mesh segmentation algorithm (such as a triangulation algorithm, a Marching Cubes algorithm); for each micro face element, records its vertex coordinates, normal vector and other geometric information to form micro face element segmentation data.

[0092] Step S32: texture mapping the microfacet segmentation data according to the heterogeneous material voxel data, thereby obtaining microfacet texture data;

[0093] The embodiment of the present invention uses a texture mapping tool or library (such as a UV mapping tool, an OpenGL texture mapping function) to perform a texture mapping operation, prepares heterogeneous material voxel data, including texture information on the surface of an object; maps the texture information in the heterogeneous material voxel data to each microfacet of the microfacet segmentation data, and methods such as UV mapping or projection texture mapping may be used; and records information such as texture coordinates and corresponding texture pixel values ​​on each microfacet to form microfacet texture data.

[0094] Step S33: performing optical property analysis on each microfacet in the microfacet texture data, thereby obtaining microfacet optical property data;

[0095] The embodiment of the present invention uses a computer vision library or a computer graphics library to perform optical property analysis (such as color analysis and reflectivity analysis) to prepare microfacet texture data, including texture coordinates and texture pixel values ​​of each microfacet; analyzes the texture pixel values ​​of each microfacet to extract optical property information, such as color, reflectivity, etc.; and records the optical property information of each microfacet to form microfacet optical property data.

[0096] Step S34: constructing a polarization metric vector field at the microfacet level according to the heterogeneous material voxel data and the microfacet optical property data, thereby obtaining polarization metric vector field data;

[0097] The embodiment of the present invention uses a computer graphics library, an optical simulation tool or an optical analysis tool to calculate the polarization metric vector field, prepares voxel data of heterogeneous materials and optical property data of microfacets; calculates the polarization metric vector field of each microfacet based on the optical properties of the heterogeneous materials and the optical properties of the microfacets, and can use an optical simulation algorithm or a physical model for calculation.

[0098] Step S35: fusing the polarization metric vector field data, the microfacet optical property data, and the three-dimensional geometric grid model to construct a polarization three-dimensional grid model.

[0099] The embodiment of the present invention uses a computer graphics library (such as OpenGL, Three.js) to perform data fusion and build a three-dimensional mesh model; prepare polarization metric vector field data, microfacet optical property data, and three-dimensional geometric mesh model data; and fuse the polarization metric vector field data and microfacet optical property data with the three-dimensional geometric mesh model. A common method is to interpolate at each vertex of the three-dimensional geometric mesh model, calculate and update the properties of the vertex using the polarization metric vector field data and microfacet optical property data; and regenerate the three-dimensional geometric mesh model based on the fused data, which includes the polarization metric vector field and optical property information.

[0100] The present invention divides the three-dimensional geometric mesh model into tiny triangular facets to provide a more fine-grained representation of the object surface. Each microfacet corresponds to a tiny area on the object surface. Through the microfacet segmentation data, geometric information such as the vertex coordinates and normal vector of each microfacet can be obtained. This microfacet segmentation data form is helpful for subsequent more detailed analysis and processing of the object surface. Texture mapping is performed on the microfacet segmentation data according to the heterogeneous material voxel data, and the texture information can be associated with the microfacet. Through texture mapping, the microfacet texture data can provide the texture information corresponding to each microfacet, so that the three-dimensional geometric mesh model has a more realistic and detailed appearance. Optical property analysis of the microfacet texture data can obtain optical properties about the microfacet. These properties may include information such as reflectivity, refractive index, transparency, etc. Through optical property analysis, the optical behavior of the object can be better simulated and expressed, making the three-dimensional geometric mesh model more realistic and real during rendering and visualization. The polarization characteristics of microfacets can be described by constructing polarization metric vector fields at the microfacet level based on heterogeneous material voxel data and microfacet optical property data. Polarization metric vector field data can provide information about the polarization state and polarization direction of microfacets, which is of great significance for simulating and analyzing the polarization effect of light and helps to achieve more realistic optical effects and rendering results. A polarization 3D mesh model can be constructed by fusing polarization metric vector field data, microfacet optical property data, and a 3D geometric mesh model. This model integrates geometric information, texture information, optical property information, and polarization metric information, and can more comprehensively describe the appearance and optical properties of objects. The polarization 3D mesh model can be applied to computer graphics, optical simulation, materials science, and other fields to provide more realistic and accurate rendering, simulation, and analysis results.

[0101] Preferably, the segmentation of the facets in step S31 includes the following process:

[0102] The thresholds of the maximum and minimum sizes of triangular face elements are dynamically adjusted according to the scale and curvature changes of the three-dimensional geometric mesh model, and vertices and edges are shared between adjacent face elements. The initial triangular mesh in the smooth surface area of ​​the three-dimensional geometric mesh model is subdivided through the Loop subdivision surface, and features are retained in the high curvature area.

[0103] The embodiment of the present invention uses a computer graphics library (such as OpenGL, Three.js) to process and segment a geometric mesh model, prepares data of a three-dimensional geometric mesh model, including vertex coordinates and face information; for each face, calculates its scale and curvature information. The scale can be obtained by calculating the bounding box or diameter of the face, and the curvature can be calculated using a curvature estimation algorithm (such as the mean curvature method, the Gaussian curvature method); according to the changes in scale and curvature, dynamically adjust the maximum and minimum size thresholds of the triangular face element. A common method is to calculate an adaptive threshold based on the scale and curvature of the face, such as using a linear function or an exponential function to calculate the threshold, so that a larger face element is generated in an area with a larger scale and a smaller curvature, and a smaller face element is generated in an area with a smaller scale and a larger curvature; according to the dynamically adjusted threshold, the three-dimensional geometric mesh model is segmented into a number of tiny triangular face elements to ensure that vertices and edges are shared between adjacent face elements. Use computer graphics libraries (such as OpenGL, Three.js) to perform loop subdivision surface operations, prepare data of tiny triangular facets obtained through facet segmentation, and use the tiny triangular facets obtained through facet segmentation as the initial triangular mesh; perform loop subdivision surface operations on the initial triangular mesh, which retains features in smooth surface areas. The loop subdivision algorithm is a commonly used subdivision surface algorithm, which generates new vertices and facets by iteratively subdividing each triangle to obtain a smoother surface. In the loop subdivision process, high curvature areas can be specially processed according to curvature information to retain their features. For example, in high curvature areas, the number of iterations of the subdivision algorithm can be controlled or the subdivision weights can be adjusted to retain more details.

[0104] The present invention dynamically adjusts the thresholds of the maximum and minimum sizes of triangular face elements according to the scale and curvature changes of the three-dimensional geometric mesh model, and can adaptively adjust the size of the triangular face elements according to the local features of the model. Through the dynamic threshold, smaller triangular face elements can be used in areas with rich details to better capture subtle surface details. In flat or smooth areas, larger triangular face elements can be used to reduce the number of face elements and improve computing efficiency. This adaptive adjustment can strike a balance between details and computing efficiency, making the three-dimensional geometric mesh model more accurate and efficient. The advantage of sharing vertices and edges between adjacent face elements is that redundant data and storage overhead are reduced. By sharing vertices and edges, the size of mesh data can be effectively reduced, saving memory and computing resources. At the same time, sharing edges and vertices also helps to maintain the continuity and smoothness of the model, making the surface more natural and seamless at the connection. By subdividing the initial triangular mesh of the smooth surface area in the three-dimensional geometric mesh model through the Loop subdivision surface, the characteristics and smoothness of the surface can be better retained. Loop subdivision is a commonly used subdivision algorithm that increases the detail level of the mesh by subdividing each triangle into a group of new triangles. In high curvature areas, Loop subdivision can divide triangles more finely to maintain the details of the surface; in low curvature areas, fewer subdivisions can be used to reduce the number of facets. This subdivision algorithm can effectively control the number of facets and computational complexity while maintaining the smoothness of the surface. Combining the above steps, a three-dimensional geometric mesh model with dynamic adjustment, shared edges and vertices, and Loop subdivision can be obtained. This model has high-precision geometric representation, low data redundancy and storage overhead, and can maintain the details and smoothness of the surface. This is very beneficial to fields such as computer graphics, object modeling, virtual reality, and animation, and can provide more realistic, detailed, and efficient three-dimensional model representation and processing.

[0105] Preferably, step S33 includes the following steps:

[0106] Step S331: extracting the RGB color and reflectivity of the voxel corresponding to each microfacet according to the heterogeneous material voxel data, thereby obtaining optical property prior data;

[0107] The embodiment of the present invention uses a computer graphics library (such as OpenGL, OpenCV) to process and analyze voxel data, prepare heterogeneous material voxel data, including the color and reflectivity information of each voxel; sample or interpolate the voxels of each microfacet, extract the RGB color value of the corresponding voxel, and sample or interpolate the voxels of each microfacet to extract the reflectivity value of the corresponding voxel. The reflectivity can be expressed as a value between 0 and 1, reflecting the degree of absorption and reflection of light by the object.

[0108] Step S332: Performing network regression processing according to the microfacet texture data and optical property prior data, thereby obtaining preliminary optical property value data;

[0109] The embodiment of the present invention uses a deep learning framework (such as TensorFlow, PyTorch) to perform network regression processing, prepares texture data and optical property prior data of microfacets, including color and reflectivity; designs an appropriate neural network structure, takes texture data as input, and predicts the optical property values ​​of microfacets; uses known texture data and corresponding optical property prior data to perform network training, and optimizes network parameters through a regression loss function, so that the network can accurately predict the optical property values ​​of microfacets.

[0110] Step S333: capturing microfacet correlation of microfacet texture data through an attention mechanism, and performing attention enhancement on preliminary optical property value data, thereby obtaining optical property value data;

[0111] The embodiment of the present invention uses a deep learning framework (such as TensorFlow, PyTorch) to implement an attention mechanism, prepares texture data and preliminary optical property value data of microfacets; uses an attention mechanism (such as a self-attention mechanism, a channel attention mechanism) to process the texture data of microfacets to capture the correlation between microfacets; applies the weight obtained by the attention mechanism to the preliminary optical property value data to enhance the optical property values ​​related to the texture data, and obtain more accurate optical property value data.

[0112] Step S334: coupling the optical property value data with a preset BRDF model to obtain microfacet optical property data, wherein the microfacet optical property data includes reflectivity, roughness, metallicity, and glossiness.

[0113] The embodiment of the present invention uses a computer graphics library (such as OpenGL, Mitsuba) to couple and calculate the BRDF model to prepare attention-enhanced optical property value data; selects an appropriate BRDF model (such as Lambertian, Cook-Torrance) as a preset model, couples the optical property value data with the selected BRDF model, and obtains the optical property data of the microfacet by calculation, including reflectivity, roughness, metallicity and glossiness.

[0114] The present invention extracts the RGB color and reflectivity of the voxels corresponding to each microfacet according to the voxel data of heterogeneous materials, and can obtain the color information and reflectivity prior data of the object surface at different positions. These data can be obtained by sampling and analyzing the voxel data, and provide a preliminary estimate of the optical properties of the microfacet. The optical property prior data provides a basis for subsequent optical property estimation and regression. The optical properties of the microfacet can be estimated by a machine learning method by using the microfacet texture data and the optical property prior data for network regression processing. By training the regression model, the preliminary optical property value can be predicted according to the texture information of the microfacet and the prior optical property data. This regression processing can infer the optical property according to the input texture data and the prior information, and provide a preliminary estimate of the optical property of the microfacet. The microfacet texture data is captured by the attention mechanism, and the preliminary optical property value data is enhanced. The attention mechanism can help the model focus on important texture features and correlations, so as to more accurately estimate the optical properties of the microfacet. Through the attention enhancement, the preliminary optical property value can be adjusted and corrected to improve the accuracy and consistency of the estimation. The optical property value data is coupled with the preset BRDF (bidirectional reflectance distribution function) model to obtain the microfacet optical property data. The BRDF model describes the reflection characteristics of light on the surface of an object. By combining the estimated optical property values ​​with the BRDF model, the optical property data of the microfacet can be obtained, including reflectivity, roughness, metallicity, and glossiness. Such optical property data can more accurately describe the optical properties of the surface of an object and provide a basis for subsequent rendering and visualization processes. Combining the above steps, by extracting heterogeneous material voxel data, network regression processing, attention mechanism and BRDF model coupling, more accurate and detailed microfacet optical property data can be obtained. These data are very useful for applications such as realistic rendering, lighting simulation, and material analysis, and can provide more realistic, credible, and lifelike optical effects and performances.

[0115] Preferably, step S4 comprises the following steps:

[0116] Step S41: constructing a representation decoupling network based on a variational autoencoder according to the polarization three-dimensional grid model, and encoding the polarization three-dimensional grid model using the representation decoupling network to obtain a potential representation vector;

[0117] The embodiment of the present invention uses a deep learning framework (such as TensorFlow, PyTorch) to construct and train a representation decoupling network, prepares polarized three-dimensional mesh model data, and designs a network structure based on a variational autoencoder (VAE) to encode the polarized three-dimensional mesh model and obtain a potential representation vector. VAE includes an encoder network and a decoder network, which learns the potential representation of the data by minimizing the reconstruction error and the regularization term of the latent space; the representation decoupling network is trained using the polarized three-dimensional mesh model data, and the potential representation vector is learned by optimizing the loss function of VAE.

[0118] Step S42: explicitly modeling different semantic features of the latent representation vector to obtain a multi-semantic feature representation;

[0119] The embodiment of the present invention uses a deep learning framework (such as TensorFlow, PyTorch) to model multi-semantic feature representation and prepare latent representation vector data; the different semantic features of the latent representation vector are explicitly modeled, and feature extraction and representation learning can be performed using fully connected layers, convolutional neural networks and other methods to obtain multi-semantic feature representation.

[0120] Step S43: decoupling inherent properties of the multi-semantic feature representation to the maximum extent based on adversarial training, thereby obtaining an inherent semantic feature tensor;

[0121] The embodiment of the present invention uses a deep learning framework (such as TensorFlow, PyTorch) to perform adversarial training and decouple inherent property learning, and designs an adversarial network, including a discriminator network and a generator network. The discriminator network is used to distinguish between the real multi-semantic feature representation and the feature representation generated by the generator network; the generator network minimizes the adversarial loss so that the generated feature representation can deceive the discriminator, thereby achieving the purpose of decoupling inherent properties.

[0122] Step S44: constructing a molecular orbital projection network, and projecting the inherent semantic feature tensor into multiple subspaces, thereby obtaining projection vector data of each semantic attribute in the subspace;

[0123] The embodiment of the present invention uses a deep learning framework (such as TensorFlow, PyTorch) to construct and train a molecular orbital projection network, prepares the intrinsic semantic feature tensor data, and designs a molecular orbital projection network for projecting the intrinsic semantic feature tensor into multiple subspaces. This can be achieved through methods such as convolutional neural networks and fully connected layers.

[0124] Step S45: performing object instance segmentation according to the projection vector data, thereby obtaining segmentation mask data of each instance;

[0125] The embodiment of the present invention uses an image segmentation algorithm and a deep learning framework (such as Mask R-CNN, UNet) to perform object instance segmentation and prepare projection vector data and polarized three-dimensional grid model data; uses the object instance segmentation algorithm to process the projection vector data, generates segmentation mask data for each instance, and segments the object instance from the background.

[0126] Step S46: performing reasoning of the occluded object part based on a single instance according to the segmentation mask data and the polarized three-dimensional grid model, thereby obtaining a single-instance three-dimensional model set;

[0127] An embodiment of the present invention uses an object part reasoning algorithm and a deep learning framework (such as Mask R-CNN, FCN) for reasoning to prepare segmentation mask data and polarized three-dimensional mesh model data; uses the object part reasoning algorithm, combined with the segmentation mask data and the polarized three-dimensional mesh model data, to reason about parts of a single instance to obtain information about the occluded object part.

[0128] Step S47: integrating the projection vector data and the single instance three-dimensional model set to obtain a molecular orbital-like topology map.

[0129] The embodiment of the present invention uses graph theory algorithms and data processing libraries (such as NetworkX and Matplotlib) to integrate and visualize information, prepare projection vector data and a single-instance three-dimensional model set; combine the projection vector data and the single-instance three-dimensional model set to obtain complete object instance information, which can be achieved by associating the projection vector with the corresponding single-instance three-dimensional model, or by calculating the similarity between them; based on the integrated information, construct a molecular orbital-like topological map, which is a graph structure composed of nodes and edges, wherein nodes represent object instances or semantic features, and edges represent the relationship between them. The connection relationship between nodes can be determined based on the similarity between object instances, semantic associations, etc., and then a graph theory algorithm is used to construct and analyze the topological map.

[0130] The present invention can convert a polarized three-dimensional grid model into a potential representation vector by using a variational autoencoder and a representation decoupling network. This encoding process helps to reduce the dimension of the data and extract important features hidden in the data. The potential representation vector can be used as the input of subsequent steps while retaining the key information of the original model. The potential representation vector is explicitly modeled and can be decomposed into different semantic feature representations. This decomposition helps to understand and explain different semantic attributes in the data, such as shape, texture, etc. Through multi-semantic feature representation, different aspects of the data can be better represented and manipulated, and the flexibility and accuracy of subsequent steps can be improved. Through adversarial training, the inherent attributes in the multi-semantic feature representation can be decoupled to the maximum extent. This means that different semantic features can be learned and represented independently of each other, avoiding redundancy and mutual influence between them. The acquisition of the inherent semantic feature tensor makes different semantic attributes more independent and controllable in the representation, providing more accurate and reliable input for subsequent steps. By constructing a molecular orbital projection network, the inherent semantic feature tensor can be projected into multiple subspaces. These subspaces represent the range of variation and feature distribution of different semantic attributes. By projecting the semantic attributes into the subspace, projection vector data describing each attribute can be obtained, which helps to analyze and process the attributes in a more fine-grained manner. By using the projection vector data for object instance segmentation, different instances can be separated from the overall model and a segmentation mask for each instance can be generated. Such segmentation mask data provides an independent representation for each instance, so that subsequent steps can process and reason about each instance individually. By using the segmentation mask data and the polarized 3D mesh model, reasoning about the object part occluded by the single-instance hydrology can be performed; this means that the occluded object part can be inferred based on the segmentation mask, thereby generating a 3D model set of a single instance; this step helps to recover the object part that cannot be observed due to hydrological occlusion, providing a more complete and accurate single-instance model. The projection vector data and the single-instance 3D model set are integrated to generate a result similar to a molecular orbital topology map; by integrating different sources of information, a more comprehensive and integrated understanding can be obtained, generating a topology map with structure and association; the result of this step helps to further analyze and apply the characteristics of the 3D model and provides insights into the overall structure of the data. The present invention converts the polarized three-dimensional grid model into a potential representation vector, obtains a multi-semantic feature representation, decouples the inherent properties of the multi-semantic feature representation, obtains the projection vector data of each semantic attribute in the subspace, realizes object instance segmentation, infers the occluded object part, and integrates information to generate a molecular orbital topology map. The combination of these steps makes it possible to better understand and process the polarized three-dimensional grid model, extract and express key features, and generate results with structure and semantics, providing a useful basis for further application and analysis.

[0131] Preferably, step S46 includes the following steps:

[0132] Step S461: segmenting the polarization three-dimensional grid model into individual instances according to the segmentation mask data, thereby obtaining a three-dimensional dataset of instance objects;

[0133] The embodiment of the present invention uses an image segmentation algorithm and a 3D model processing library (such as MeshLab, Open3D) to perform instance segmentation and three-dimensional data set processing; segmentation mask data is used to segment the polarized three-dimensional mesh model, and each instance is extracted to form a three-dimensional data set of instance objects. Image segmentation algorithms (such as Mask R-CNN) or methods based on geometric information (such as methods based on region growing or clustering) can be used for segmentation.

[0134] Step S462: extracting the depth value and normal vector of each voxel of the voxel data of each instance in the three-dimensional data set of the instance object, and performing depth value discontinuous jump area detection based on the viewing direction, so as to obtain occlusion boundary data;

[0135] The embodiment of the present invention uses a three-dimensional geometry processing library (such as Open3D, PointCloudLibrary) to extract depth values ​​and normal vectors and detect jump areas, voxelizes each instance in the three-dimensional data set of the instance object, and calculates the depth value and normal vector of each voxel; then, based on the viewing direction, detects areas where the depth value jumps discontinuously, which may represent occlusion boundaries. The jump areas can be detected by comparing the depth value differences between adjacent voxels and marked as occlusion boundaries.

[0136] Step S463: performing region growing according to the occlusion boundary data, thereby obtaining preliminary occlusion mask data;

[0137] The embodiment of the present invention uses a region growing algorithm and an image processing library (such as OpenCV) to perform region growing, uses occlusion boundary data as seed points, and uses the region growing algorithm to start from the seed points and gradually grow the expansion area until a certain stop condition is reached. In this way, preliminary occlusion mask data can be obtained, in which the occluded area is marked as the foreground and the non-occluded area is marked as the background.

[0138] Step S464: performing vector abnormal change analysis based on the preliminary occlusion mask data and its corresponding normal vector, and correcting the preliminary occlusion mask data to obtain corrected occlusion mask data;

[0139] The embodiment of the present invention uses an outlier detection algorithm and a statistical analysis library (such as Scikit-learn and NumPy) to analyze and correct abnormal changes in vectors, and performs abnormal change analysis of vectors based on preliminary occlusion mask data and its corresponding normal vector. The abnormally changed normal vector may indicate the inconsistency of the occlusion area. An outlier detection algorithm (such as Isolation Forest and One-Class SVM) can be used to identify abnormal normal vectors, and the preliminary occlusion mask data can be corrected according to the abnormal situation to further improve the accuracy of the occlusion area.

[0140] Step S465: performing shape analysis on the connected regions according to the corrected occlusion mask data, and removing regions that are too small / too large, thereby obtaining occlusion region mask data of a single instance object, wherein the 5% quantile of the number of voxels is taken as the lower threshold, and the 95% quantile of the number of voxels is taken as the upper threshold;

[0141] The embodiment of the present invention uses an image processing library and morphological operations (such as OpenCV) to perform connected region analysis and region screening, and uses image processing technology to perform connected region analysis based on the corrected occlusion mask data, and marks the connected occlusion region as the occlusion region mask data of a single instance object; then, a shape analysis is performed based on the number of voxels in the region, and regions that are too small or too large are removed; the number of voxels in all regions is calculated, and the region with a voxel number lower than the 5% quantile is marked as the background, and the region with a voxel number higher than the 95% quantile is marked as the abnormal region, and the region with a voxel number between the 5% quantile and the 95% quantile is retained as the occlusion region mask data of a single instance object.

[0142] Step S466: Use the generative adversarial network to infer the occluded object part of the occluded area mask data, so as to obtain a single instance three-dimensional model set.

[0143] The embodiment of the present invention uses a generative adversarial network (GAN) and a deep learning framework (such as TensorFlow, PyTorch) to perform reasoning on occluded objects and reconstruct three-dimensional models. The generative adversarial network is used to take the mask data of the occluded area as input to perform reasoning on the occluded object part. The generative adversarial network can learn the characteristics of the occluded area and generate an estimate of the occluded object part. In this way, a single-instance three-dimensional model set can be obtained.

[0144] The present invention can separate different instances in the entire model by segmenting the polarized three-dimensional grid model into individual instances according to the segmentation mask data, and obtain an independent three-dimensional data set for each instance; this segmentation allows the subsequent steps to perform individual processing and analysis on each instance, thereby improving the understanding and application of instance-level features. By extracting the depth value and normal vector of each voxel in the three-dimensional data set of the instance object, and performing a depth value discontinuous jump zone detection based on the viewing direction, the occlusion boundary between the instance objects can be determined; this helps to identify the occlusion relationship between different instances, provides useful information about the occlusion structure, and provides a basis for occlusion processing and reasoning in subsequent steps. By performing regional growth based on occlusion boundary data, adjacent occlusion boundary points can be connected and expanded into a continuous occlusion area, so that preliminary occlusion mask data can be obtained, so that the occluded area is clearly represented in the data, providing a basis for occlusion analysis and correction in subsequent steps. By performing vector abnormal change analysis on the preliminary occlusion mask data and its corresponding normal vector, abnormal changes around the occlusion boundary can be detected. Based on these abnormal changes, the preliminary occlusion mask data can be corrected to improve the accuracy and consistency of the occlusion mask; the corrected occlusion mask data better reflects the true occlusion relationship between instances, providing more reliable input for subsequent steps. By performing shape analysis of connected regions on the corrected occlusion mask data, different occlusion regions of instance objects can be identified, and regions that are too small or too large can be removed, so that more accurate occlusion region mask data of a single instance object can be obtained, limiting the occlusion relationship within a reasonable range. By setting lower and upper thresholds, the occlusion region can be appropriately filtered according to the number of voxels of the instance, ensuring that the extracted occlusion region is neither too small nor too enlarged, thereby obtaining more accurate results. By using a generative adversarial network to reason about the mask data of the occluded area, the shape and structure of the occluded object part can be inferred. The generative adversarial network can generate a reasonable object part within the occluded area by learning the relationship between the known occluded area and other visible areas; in this way, a complete three-dimensional model set of a single instance can be obtained, including the occluded part, providing more comprehensive instance information and data, which is of great help for subsequent target recognition, reconstruction or other tasks. The present invention improves the understanding of instance objects, the extraction of occlusion relationships and the accuracy of model reconstruction through improvements in instance segmentation, occlusion boundary detection, occlusion mask generation, occlusion correction, occlusion area screening and reasoning. The combination and sequential design of these steps help to extract the complete information of a single instance from the polarized three-dimensional grid model and generate a three-dimensional model set containing the occluded part.

[0145] Preferably, step S5 comprises the following steps:

[0146] Step S51: Retrieving relevant subgraphs and semantic attributes from the molecular orbital-like topological graph according to a preset target, thereby obtaining subgraph semantic attribute data;

[0147] The embodiment of the present invention uses graph theory algorithms and graph analysis libraries (such as NetworkX, igraph) to process and analyze graph data; first, define relevant search rules and constraints according to preset goals, extract relevant subgraphs from the molecular orbital-like topological graph, and use graph theory algorithms (such as depth-first search, breadth-first search) to search for subgraphs in the topological graph; then, extract corresponding semantic attributes according to the structure and characteristics of the subgraph, which may include node labels, edge types, connection relationships between nodes, etc., thereby obtaining subgraph semantic attribute data.

[0148] Step S52: Designing a targeted recognition model based on the attention mechanism for the sub-graph semantic attribute data, thereby obtaining an attention recognition model;

[0149] The embodiment of the present invention uses a deep learning framework (such as TensorFlow, PyTorch) for model design and training. Based on the sub-graph semantic attribute data, a targeted recognition model based on the attention mechanism is designed and trained. Graph convolutional neural network (GCN), graph attention network (GAT) and other graph neural network models can be used as basic models. In the model design, the introduction of the attention mechanism can make the model pay more attention to the sub-graph features related to the preset target. By training the model, it can learn effective feature representation and recognition capabilities, thereby obtaining an attention recognition model.

[0150] Step S53: Accurately identify, locate and track the preset target according to the attention recognition model, so as to obtain re-identification metadata.

[0151] The embodiment of the present invention uses a computer vision library and a deep learning framework for target recognition and tracking (such as OpenCV, TensorFlow Object Detection API); uses an attention recognition model to accurately recognize and locate a preset target, and can use a target detection algorithm (such as a deep learning-based target detector) to recognize the target in an image or video frame and obtain a bounding box of the target position; then, a target tracking algorithm (such as a tracker based on Kalman filtering or optical flow) can be used to track the motion trajectory of the target in a continuous frame sequence. In this way, re-recognition metadata can be obtained, including the recognition result, location information, and motion trajectory of the target.

[0152] The present invention can obtain subgraph data related to the preset target by retrieving relevant subgraphs and semantic attributes from the molecular orbital topology map; these subgraph data contain molecular structure and chemical property information related to the target, which provides an important basis for target identification and positioning in the subsequent steps. By designing a targeted recognition model based on the attention mechanism, the key features and contextual information of the target can be learned from the subgraph semantic attribute data; the attention mechanism can make the model pay more attention to the area and attributes related to the target, improve the accuracy and robustness of target recognition, and the obtained attention recognition model can effectively locate and identify the target in complex subgraph data. By applying the attention recognition model, the preset target can be accurately identified, located and tracked. Through the attention mechanism, the model can accurately focus on the area where the target is located and extract key features for identification; therefore, this step can obtain re-identification metadata, which contains important data such as the recognition result, location information and tracking trajectory of the target. These metadata can be used for subsequent analysis, decision-making and further processing. Retrieve relevant subgraph semantic attributes from the molecular orbital topology map to provide key information about the target; design a targeted recognition model based on the attention mechanism to improve the accuracy and robustness of target recognition; accurately identify, locate and track the target based on the attention recognition model to obtain important re-identification metadata. The combination and sequential design of these steps help to extract target-related information from the molecular orbital topology map and achieve accurate identification, positioning and tracking of the target, which has important application value for molecular structure analysis, drug development and other tasks in the field of chemistry.

[0153] Therefore, the embodiments should be regarded as illustrative and non-restrictive from all points, and the scope of the present invention is limited by the appended claims rather than the above description, and it is therefore intended that all changes falling within the meaning and range of equivalent elements of the application documents are included in the present invention.

[0154] The above description is only a specific embodiment of the present invention, so that those skilled in the art can understand or implement the present invention. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but should conform to the widest scope consistent with the principles and novel features invented herein.

Claims

1. A method for automatically capturing and acquiring material weighing images, characterized in that: The following steps are involved: Step S1: collect information on on-site materials and vehicles based on multiple sensors, and perform point cloud registration processing based on a spatiotemporal coordinate system to obtain heterogeneous material voxel data, wherein the multiple sensors include structured light vision sensors, lidar sensors, and high-resolution camera sensors. Step S1 includes: Step S11: projecting a coded light field onto on-site materials and vehicles through a structured light vision sensor, and capturing reflected coded light, thereby obtaining depth image data, wherein the structured light vision sensor is composed of a projector and a camera; Step S12: using a laser radar sensor to emit laser beams to on-site materials and vehicles, measure the time difference of reflected light, and calculate the distance, thereby obtaining three-dimensional point cloud data; Step S13: obtaining high-definition RGB image data of the scene through a high-resolution camera sensor; Step S14: performing spatiotemporal registration processing on the depth image data and the three-dimensional point cloud data based on the spatiotemporal coordinate system, thereby generating structured light point cloud data; Step S15: projecting the structured light point cloud data into a unified three-dimensional grid voxel, thereby obtaining three-dimensional grid voxel data, wherein the voxel includes a geometric position and a surface normal vector; Step S16: performing statistical coloring of each voxel of the three-dimensional grid voxel data according to the RGB image data, thereby obtaining heterogeneous material voxel data; Step S2: Perform visual context analysis on the heterogeneous material voxel data to obtain a visual context feature map; perform depth estimation based on structured light vision on the visual context feature map, and perform voxel-to-voxel ray tracing to obtain a three-dimensional geometric mesh model. Step S2 includes: Step S21: construct an encoder-decoder network based on 3D convolution, input the heterogeneous material voxel data into the encoder-decoder network, extract local and global context features layer by layer through the 3D convolution of the encoder, and obtain context encoding feature data; Step S22: Perform visual context analysis on the context encoding feature data through a decoder to obtain a visual context feature map, wherein the decoder includes a semantic segmentation branch, a depth estimation branch, and an illumination analysis branch. Step S22 includes: Step S221: performing pixel-level semantic segmentation based on different entities on the context encoding feature data through the semantic segmentation branch, thereby obtaining a semantic segmentation voxel atlas; Step S222: using the depth estimation branch to estimate the depth value of each voxel of the context encoding feature data, thereby obtaining a high-precision depth voxel map; Step S223: performing illumination distribution analysis of the shadow degree and reflection intensity of each voxel on the context encoding feature data through the illumination analysis branch, thereby obtaining an illumination voxel atlas; Step S224: accurately aligning the semantic segmentation voxel map, the high-precision depth voxel map, and the illumination voxel map in space and semantics, and fusing them into a visual context feature map; Step S23: using the depth image data as a priori, fusing the context features and the depth priori into the visual context feature map, and performing depth value regression to obtain a high-precision depth voxel map; Step S24: using the depth voxel map as the distance field to perform back-projection of each voxel in the three-dimensional space, and emitting sight tracking one by one from the camera perspective until hitting the surface of other voxels, calculating the three-dimensional coordinates and surface normal vector of the hitting point, thereby obtaining sight tracking result data; Step S25: construct triangular facets according to adjacent normal vectors in the line of sight tracking result data, and perform facet refinement based on computer graphics smoothing technology to obtain a three-dimensional geometric mesh closed model without holes; Step S26: performing Laplace smoothing on the three-dimensional geometric mesh closed model, and performing detail enhancement according to the visual context feature map, thereby obtaining a three-dimensional geometric mesh model; Step S3: Segment the three-dimensional geometric grid model into a number of tiny geometric facets to obtain micro-facet segmentation data; perform optical property analysis on the micro-facet segmentation data according to the heterogeneous material voxel data, and perform data fusion with the three-dimensional geometric grid model to obtain a polarized three-dimensional grid model. Step S3 includes: Step S31: Segment the three-dimensional geometric mesh model into a number of tiny triangular facets, each of which corresponds to a tiny area on the surface of the object, thereby obtaining microfacet segmentation data, wherein the microfacet segmentation data includes geometric information of vertices and normal vectors of each microfacet; Step S32: texture mapping the microfacet segmentation data according to the heterogeneous material voxel data, thereby obtaining microfacet texture data; Step S33: performing optical property analysis on each microfacet in the microfacet texture data, thereby obtaining microfacet optical property data. Step S33 includes: Step S331: extracting the RGB color and reflectivity of the voxel corresponding to each microfacet according to the heterogeneous material voxel data, thereby obtaining optical property prior data; Step S332: Performing network regression processing according to the microfacet texture data and optical property prior data, thereby obtaining preliminary optical property value data; Step S333: capturing microfacet correlation of microfacet texture data through an attention mechanism, and performing attention enhancement on preliminary optical property value data, thereby obtaining optical property value data; Step S334: coupling the optical property value data with a preset BRDF model to obtain microfacet optical property data, wherein the microfacet optical property data includes reflectivity, roughness, metallicity, and glossiness; Step S34: constructing a polarization metric vector field at the microfacet level according to the heterogeneous material voxel data and the microfacet optical property data, thereby obtaining polarization metric vector field data; Step S35: fusing the polarization metric vector field data, the microfacet optical property data, and the three-dimensional geometric grid model to construct a polarization three-dimensional grid model; Step S4: using a variational autoencoder to perform visual decoupling of the intrinsic representation of the polarized three-dimensional grid model, thereby obtaining an intrinsic semantic feature tensor; projecting the intrinsic semantic feature tensor into a molecular orbital-like subspace, and performing three-dimensional reconstruction based on instance segmentation and occlusion region reasoning, thereby obtaining a molecular orbital-like topological map, step S4 includes: Step S41: constructing a representation decoupling network based on a variational autoencoder according to the polarization three-dimensional grid model, and encoding the polarization three-dimensional grid model using the representation decoupling network to obtain a potential representation vector; Step S42: explicitly modeling different semantic features of the latent representation vector to obtain a multi-semantic feature representation; Step S43: decoupling inherent properties of the multi-semantic feature representation to the maximum extent based on adversarial training, thereby obtaining an inherent semantic feature tensor; Step S44: constructing a molecular orbital projection network, and projecting the inherent semantic feature tensor into multiple subspaces, thereby obtaining projection vector data of each semantic attribute in the subspace; Step S45: performing object instance segmentation according to the projection vector data, thereby obtaining segmentation mask data of each instance; Step S46: performing reasoning based on the occluded object part of a single instance according to the segmentation mask data and the polarized three-dimensional grid model, thereby obtaining a single-instance three-dimensional model set. Step S46 includes: Step S461: segmenting the polarization three-dimensional grid model into individual instances according to the segmentation mask data, thereby obtaining a three-dimensional dataset of instance objects; Step S462: extracting the depth value and normal vector of each voxel of the voxel data of each instance in the three-dimensional data set of the instance object, and performing depth value discontinuous jump area detection based on the viewing direction, so as to obtain occlusion boundary data; Step S463: performing region growing according to the occlusion boundary data, thereby obtaining preliminary occlusion mask data; Step S464: performing vector abnormal change analysis based on the preliminary occlusion mask data and its corresponding normal vector, and correcting the preliminary occlusion mask data to obtain corrected occlusion mask data; Step S465: performing shape analysis on the connected regions according to the corrected occlusion mask data, and removing regions that are too small / too large, thereby obtaining occlusion region mask data of a single instance object, wherein the 5% quantile of the number of voxels is taken as the lower threshold, and the 95% quantile of the number of voxels is taken as the upper threshold; Step S466: using a generative adversarial network to infer the occluded object part of the occluded area mask data, thereby obtaining a single instance three-dimensional model set; Step S47: integrating the projection vector data and the single instance three-dimensional model set to obtain a molecular orbital-like topology map; Step S5: accurately locate and identify vehicles and materials according to the molecular orbital topology map and the preset target, so as to obtain re-identification metadata. Step S5 includes: Step S51: Retrieving relevant subgraphs and semantic attributes from the molecular orbital-like topological graph according to a preset target, thereby obtaining subgraph semantic attribute data; Step S52: Designing a targeted recognition model based on the attention mechanism for the sub-graph semantic attribute data, thereby obtaining an attention recognition model; Step S53: Accurately identify, locate and track the preset target according to the attention recognition model, so as to obtain re-identification metadata.

2. The automatic capture and acquisition method based on material weighing image screen according to claim 1 is characterized in that: The segmentation of the facets in step S31 includes the following process: The thresholds of the maximum and minimum sizes of triangular face elements are dynamically adjusted according to the scale and curvature changes of the three-dimensional geometric mesh model, and vertices and edges are shared between adjacent face elements; The initial triangular mesh in the smooth surface area of ​​the 3D geometric mesh model is subdivided through the Loop subdivision surface, and features are retained in the high curvature area.

Images