Automatic dinner plate dish identification and volume estimation system based on RGB-D vision

Abstract

The invention discloses a dinner plate dish automatic identification and volume estimation system based on RGB-D vision, and belongs to the technical field of RGB-D vision. Comprising a multi-modal data acquisition and preprocessing module, a dinner plate sensing stability judgment module, a dish instance segmentation module, a space coordinate inversion point cloud generation module, a dish geometric model reconstruction module, a multi-strategy adaptive volume calculation module and a data output module. Volume estimation is converted from plane approximation to direct calculation based on real space geometric morphology; through construction and processing of the three-dimensional point cloud of the dishes, the system can better deal with complex conditions of dish stacking, uneven thickness, placement posture change and the like, errors caused by a shooting angle, illumination change and visual occlusion are remarkably reduced, and a volume estimation result is more stable and reliable.

Description

Technical Field

[0001] This invention belongs to the field of RGB-D vision technology, specifically referring to an automatic identification and volume estimation system for dishes on plates based on RGB-D vision. Background Technology

[0002] Currently, in scenarios such as dietary assessment, nutrition monitoring, and smart canteen management, the accurate identification of food items on plates and the estimation of intake still heavily rely on manual weighing or recording. Traditional methods are not only time-consuming and labor-intensive, making it difficult to adapt to continuous operations in high-traffic scenarios, but they are also easily affected by subjective judgment, making it difficult to guarantee the consistency and traceability of data.

[0003] With the development of depth cameras and visual algorithms, people have begun to try to use images to automatically detect food and estimate intake. However, most existing solutions remain at the stage of area estimation based on 2D RGB images, and can only indirectly infer volume through contours and projected areas. They are insufficient in characterizing the 3D structural information such as the stacking height and thickness variation of dishes, resulting in large volume errors. A few works have introduced 3D reconstruction ideas, but they usually rely on complex structured light scanning, rotating tables or multi-view acquisition devices, which are costly and complex to deploy, making them unsuitable for large-scale promotion in real-world application environments such as school cafeterias. The food segmentation relies solely on RGB image features and lacks effective fusion of depth information, resulting in blurred segmentation boundaries and serious recognition confusion in complex scenarios such as food edges, overlapping areas and soup edges. It is impossible to accurately distinguish adjacent dishes, lacks a precise coordinate transformation mechanism, and the point cloud coordinates are not accurately aligned with the segmentation mask. The initial point cloud boundaries do not match the actual shape of the dishes, resulting in poor geometric model reconstruction quality, low reliability of volume calculation, and lack of adaptability. It is impossible to dynamically select the optimal calculation strategy based on the quality of the dish's geometric model, resulting in large volume calculation errors when processing complex dishes. To address this, we propose an automatic identification and volume estimation system for dishes on plates based on RGB-D vision. Summary of the Invention

[0004] The purpose of this invention is to provide an automatic identification and volume estimation system for dishes on plates based on RGB-D vision, so as to solve the problems mentioned in the background art.

[0005] To achieve the above objectives, the present invention provides the following technical solution: an automatic identification and volume estimation system for dishes based on RGB-D vision, including a multimodal data acquisition and preprocessing module, a dish perception stability judgment module, a dish instance segmentation module, a spatial coordinate inversion point cloud generation module, a dish geometric model reconstruction module, a multi-strategy adaptive volume calculation module, and a data output module;

[0006] The multimodal data acquisition and preprocessing module acquires data through a depth camera, obtaining a color image and a depth image that are synchronously aligned with the plate area;

[0007] The plate stability judgment module detects the plate area and analyzes short-term depth changes based on the acquired color image to determine the plate stability.

[0008] The dish instance segmentation module segments dish instances based on the determined stable frames using fused RGB-D features, generating a pixel-level segmentation mask for each dish.

[0009] The spatial coordinate inversion point cloud generation module performs three-dimensional spatial coordinate inversion based on the segmentation mask and the corresponding depth image to construct the initial three-dimensional point cloud for each dish.

[0010] The food geometric model reconstruction module performs filtering, plate plane removal, and surface reconstruction based on the initial 3D point cloud to obtain an optimized food geometric model.

[0011] The multi-strategy adaptive volume calculation module performs volume calculations based on the optimized dish geometry model;

[0012] The data output module performs data standardization and encapsulation based on the dish category and volume results, and outputs the data through the API interface.

[0013] Preferably, the multimodal data acquisition and preprocessing module is wirelessly connected to the plate perception stability judgment module and the dish instance segmentation module; the plate perception stability judgment module is wirelessly connected to the dish instance segmentation module; the dish instance segmentation module is wirelessly connected to the spatial coordinate inversion point cloud generation module and the data output module; the spatial coordinate inversion point cloud generation module is wirelessly connected to the dish geometric model reconstruction module; the dish geometric model reconstruction module is wirelessly connected to the multi-strategy adaptive volume calculation module; and the multi-strategy adaptive volume calculation module is wirelessly connected to the data output module.

[0014] Preferably, the multimodal data acquisition and preprocessing module sets the exposure time, frame rate, and trigger signal of the color camera and the depth camera through the underlying driver interface of the depth camera, enables hardware-level synchronization, transforms the pixel coordinate system of the depth image to the coordinate system of the color image through the intrinsic and extrinsic parameter matrices obtained by camera calibration, performs coordinate system transformation through rotation matrix R and translation vector T, calculates the corresponding position of each pixel in the depth image in the color image through bilinear interpolation, generates a depth-color pixel mapping table, removes outlier noise points based on the local statistical characteristics of depth values, and uses a region growing-based filling algorithm to repair missing data and generate a continuous depth distribution for holes in the depth image caused by occlusion or low-texture areas, and performs timestamp alignment processing on the acquired depth image and color image.

[0015] Preferably, the plate perception stability judgment module acquires a color image, locates the plate area in the image through target detection, and outputs new coordinates within the plate bounding box, as follows:

[0016] ,

[0017] In the formula, , This indicates the center coordinates of the detected bounding box of the plate. , Indicates the center coordinates of the anchor frame. , This represents the bounding box regression offset. and These represent the width and height of the anchor frame, respectively. This indicates the ratio of the plate height to the constraint factor;

[0018] Obtain the depth values ​​of all pixels within the bounding box of the plate. By combining depth sensor data, the plate area is detected and the corresponding depth value is generated, which is achieved as follows:

[0019] ,

[0020] In the formula, This represents the weighted average depth value of the plate area. This represents the depth value of the i-th pixel within the bounding box of the plate. This represents the mean value of the depth of the plate area. The standard deviation of the depth values ​​of the plate area is represented by N, which represents the total number of pixels within the plate's bounding box.

[0021] Preferably, the plate sensing stability judgment module obtains the average depth value of consecutive frames, tracks the depth value change of the plate area, and calculates the short-term depth change, which is implemented as follows:

[0022] ,

[0023] In the formula, This represents the short-term depth change. This represents the average depth value of the current frame. M represents the average depth value of the previous frame, and M represents the number of consecutive frames. This represents the amount of depth change across consecutive frames. Indicates the normalization factor;

[0024] A preset stability threshold is set, and the short-term depth change is compared with the preset stability threshold. If the short-term depth change is lower than the threshold, the plate is determined to be in a stable state; otherwise, it is determined to be in an unstable state.

[0025] Preferably, the dish instance segmentation module acquires a stable frame color image and a depth image, concatenates the depth image as a fourth channel with the RGB image to generate a four-channel input tensor (R, G, B, D), and calculates the depth gradient map. , The gradient vector of the depth image is used to generate a depth-guided mask. , for , Represents the gradient sensitivity coefficient. This represents the gradient threshold.

[0026] Preferably, the dish instance segmentation module generates a dish force segmentation mask based on a depth-guided mask, which is implemented as follows:

[0027] ,

[0028] In the formula, This represents the segmentation mask for dish instances. Representing the features of an RGB image, Features representing depth images Let represent the deep feature enhancement coefficient, W represent the weights of the segmentation network, and b represent the bias term. This represents the Sigmoid activation function.

[0029] Preferably, the spatial coordinate inversion point cloud generation module acquires a dish instance segmentation mask and a depth image, traverses all valid pixels in the dish instance segmentation mask, extracts the corresponding depth values, and for each valid pixel coordinate (u, v) in the dish instance segmentation mask, obtains the depth value from the depth image D. Generate a list of depth values ; Pixel coordinates (focal length:) are obtained through camera intrinsic parameters , Principal point coordinates: , The depth values ​​are converted into three-dimensional spatial coordinates (X, Y, Z). For each valid pixel, (X, Y, Z) is calculated to generate an initial point cloud. The three-dimensional coordinates are organized into structured point cloud data, and a standardized initial three-dimensional point cloud data package is generated.

[0030] Preferably, the dish geometric model reconstruction module acquires an initial 3D point cloud, calculates the number of neighboring points for each point, removes outliers with fewer than a threshold neighboring points, eliminates sensor noise and isolated points, and, based on the point cloud density, removes points with fewer than a preset number of points within a radius to generate a purified point cloud. A planar model is then fitted into the purified point cloud, and the bottom plane of the plate is determined by minimizing the distance error from the point to the plane. Calculate the distance from each point to the fitted plane; input the dish point cloud into the Baisong reconstruction, and solve the implicit function. Generate a smooth and continuous triangular mesh model, adjust the reconstruction depth and resolution, perform edge detection on the reconstructed model, remove discontinuous patches, and generate the dish's geometric model.

[0031] Preferably, the multi-strategy adaptive volume calculation module acquires the dish's geometric model, calculates the angles between the normal vectors of all triangular facets of the dish's geometric model and the normal vectors outside the model, and determines that the model is closed if all angles are less than a preset angle; otherwise, it is determined to be an open model. The density threshold is adaptively adjusted according to the dish's height, and the actual volume of the model is calculated using the triangular mesh volume integral formula.

[0032] ,

[0033] In the formula, V represents the total volume of the entire geometric model of the dish. The coefficient representing the volume calculation. , , This represents the coordinate vectors of the three vertices of the triangle.

[0034] Preferably, the data output module acquires the dish type and volume results, generates a standardized data packet in a unified format, automatically selects the output method according to the system's preset configuration, verifies the validity of the API endpoint, authentication token, and queue connection status in real time, sends a standardized JSON data packet through the selected interface, and generates push logs.

[0035] Compared with the prior art, the beneficial effects of the present invention are:

[0036] 1. This invention significantly improves the automation level and accuracy of food volume estimation by constructing a complete system link; the system achieves closed-loop processing from image acquisition to volume output through multimodal data synchronous acquisition, depth-guided segmentation, 3D point cloud reconstruction and adaptive volume calculation.

[0037] 2. This invention uses a deep gradient-guided feature fusion mechanism to automatically enhance the weight of deep features in the edge region of the dish and retain RGB features in the inner region; the generation of the depth-guided mask ensures more accurate segmentation boundaries, especially in the edge regions of overlapping dishes and soup; it improves the boundary accuracy of dish instance segmentation and effectively avoids dish recognition confusion.

[0038] 3. This invention accurately calculates the three-dimensional coordinates of each pixel using camera intrinsic parameters, ensuring that the point cloud coordinates are strictly aligned with the segmentation mask; the effective use of depth values ​​and the accurate implementation of coordinate transformation make the initial point cloud boundary highly consistent with the actual shape of the dish; this module provides high-precision point cloud data for geometric model reconstruction, and the improved matching degree between the point cloud boundary and the actual shape of the dish directly determines the reliability of volume calculation, avoiding volume estimation deviations caused by coordinate transformation errors;

[0039] 4. This invention dynamically selects between surface integration and voxelization methods through model quality assessment, enabling high-fidelity models to use high-precision surface integration and low-quality models to use robust voxelization calculations; the cross-validation mechanism ensures the reliability of volume calculation results and reduces the error rate; thus improving the accuracy of volume calculation for complex dishes.

[0040] 5. This invention explicitly introduces and utilizes depth information to reconstruct the three-dimensional point cloud of the dish, transforming volume estimation from planar approximation to direct calculation based on the real spatial geometry. Through the construction and processing of the three-dimensional point cloud of the dish, the system can better cope with complex situations such as dish stacking, uneven thickness, and changes in placement posture, significantly reducing errors caused by shooting angle, lighting changes, and visual occlusion, making the volume estimation results more stable and reliable. Attached Figure Description

[0041] Figure 1 This is a schematic diagram of the automatic identification and volume estimation system for dishes on plates based on RGB-D vision according to the present invention;

[0042] Figure 2 This is the operation flow of the RGB-D vision-based automatic plate food identification and volume estimation system of the present invention. Figure 1 ;

[0043] Figure 3 This is the operation flow of the RGB-D vision-based automatic plate food identification and volume estimation system of the present invention. Figure 2 ;

[0044] Figure 4 This is the operation flow of the RGB-D vision-based automatic plate food identification and volume estimation system of the present invention. Figure 3 ;

[0045] Figure 5 This is the operation flow of the RGB-D vision-based automatic plate food identification and volume estimation system of the present invention. Figure 4 . Detailed Implementation

[0046] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0047] Example

[0048] Please see Figures 1-5 As shown, the present invention provides a technical solution including a multimodal data acquisition and preprocessing module, a plate perception stability judgment module, a dish instance segmentation module, a spatial coordinate inversion point cloud generation module, a dish geometric model reconstruction module, a multi-strategy adaptive volume calculation module, and a data output module;

[0049] The multimodal data acquisition and preprocessing module acquires data through a depth camera, obtaining a color image and a depth image that are synchronously aligned with the plate area;

[0050] The plate stability judgment module detects the plate area and analyzes short-term depth changes based on the acquired color image to determine the plate stability.

[0051] The dish instance segmentation module segments dish instances based on the determined stable frames using fused RGB-D features, generating a pixel-level segmentation mask for each dish.

[0052] The spatial coordinate inversion point cloud generation module performs three-dimensional spatial coordinate inversion based on the segmentation mask and the corresponding depth image to construct the initial three-dimensional point cloud for each dish.

[0053] The food geometric model reconstruction module performs filtering, plate plane removal, and surface reconstruction based on the initial 3D point cloud to obtain an optimized food geometric model.

[0054] The multi-strategy adaptive volume calculation module performs volume calculations based on the optimized dish geometry model;

[0055] The data output module performs data standardization and encapsulation based on the dish category and volume results, and outputs the data through the API interface.

[0056] In this embodiment, the multimodal data acquisition and preprocessing module is wirelessly connected to the plate perception stability judgment module and the dish instance segmentation module. The plate perception stability judgment module is wirelessly connected to the dish instance segmentation module. The dish instance segmentation module is wirelessly connected to the spatial coordinate inversion point cloud generation module and the data output module. The spatial coordinate inversion point cloud generation module is wirelessly connected to the dish geometric model reconstruction module. The dish geometric model reconstruction module is wirelessly connected to the multi-strategy adaptive volume calculation module. The multi-strategy adaptive volume calculation module is wirelessly connected to the data output module.

[0057] In this embodiment, the multimodal data acquisition and preprocessing module sets the exposure time, frame rate and trigger signal of the color camera and the depth camera to be completely consistent through the underlying driver interface of the depth camera, and enables the hardware-level synchronization function to ensure that the two image acquisitions are strictly aligned on the time axis, avoiding inter-frame misalignment caused by the movement of the plate or changes in ambient light.

[0058] Specifically, the intrinsic parameter matrices obtained through camera calibration, such as focal length and principal point coordinates, and the extrinsic parameter matrices, such as relative pose between cameras, are used to transform the pixel coordinate system of the depth image to the color image coordinate system. This coordinate system transformation is performed using a rotation matrix R and a translation vector T.

[0059] Specifically, the corresponding position of each pixel in the depth map in the color image is calculated by bilinear interpolation, generating a depth-color pixel mapping table. Based on the local statistical characteristics of the depth values, such as the mean ± 3 times the standard deviation, outlier noise points are removed.

[0060] For holes in the depth map caused by occlusion or low-texture areas, a region-growing-based filling algorithm is used, such as interpolating the depth values ​​of adjacent pixels to repair missing data and generate a continuous depth distribution. The acquired depth image and color image are time-stamp aligned to generate a visualization of the registration results between the depth map and the color image. The location is indicated by synchronously lighting up RGB lights to verify the data alignment effect.

[0061] In this embodiment, the plate perception stability judgment module acquires a color image, locates the plate area in the image through target detection, and outputs new coordinates within the plate's bounding box, which is implemented as follows:

[0062] ,

[0063] In the formula, , This indicates the center coordinates of the detected bounding box of the plate. , Indicates the center coordinates of the anchor frame. , This represents the bounding box regression offset. and These represent the width and height of the anchor frame, respectively. This indicates the ratio of the plate height to the constraint factor;

[0064] Specifically, obtain the depth values ​​of all pixels within the bounding box of the plate. By combining depth sensor data, the plate area is detected and the corresponding depth value is generated, which is achieved as follows:

[0065] ,

[0066] In the formula, This represents the weighted average depth value of the plate area. This represents the depth value of the i-th pixel within the bounding box of the plate. This represents the mean value of the depth of the plate area. The standard deviation of the depth values ​​of the plate area is represented by N, which represents the total number of pixels within the plate's bounding box.

[0067] In this embodiment, the plate sensing stability judgment module obtains the average depth value of continuous frames, tracks the depth value change of the plate area, calculates the short-term depth change, and quantifies the dynamic fluctuation of the plate in spatial position, which is achieved as follows:

[0068] ,

[0069] In the formula, This represents the short-term depth change. This represents the average depth value of the current frame. M represents the average depth value of the previous frame, and M represents the number of consecutive frames. This represents the amount of depth change across consecutive frames. Indicates the normalization factor;

[0070] A preset stability threshold is set, and the short-term depth change is compared with the preset stability threshold. If the short-term depth change is lower than the threshold, the plate is determined to be in a stable state; otherwise, it is determined to be in an unstable state.

[0071] In this embodiment, the dish instance segmentation module acquires a stable frame color image (RGB) and a depth image D, performs unified resolution adjustment and normalization processing, verifies the spatial alignment accuracy between the depth image and the color image, and concatenates the depth image as the fourth channel with the RGB image to generate a four-channel input tensor (R, G, B, D), and calculates the depth gradient map. , The gradient vector of the depth image is used to generate a depth-guided mask. , for , Represents the gradient sensitivity coefficient. This represents the gradient threshold.

[0072] In this embodiment, the dish instance segmentation module generates a dish force segmentation mask based on a depth-guided mask, which is implemented as follows:

[0073] ,

[0074] In the formula, This represents the segmentation mask for dish instances. Representing the features of an RGB image, Features representing depth images Let represent the deep feature enhancement coefficient, W represent the weights of the segmentation network, and b represent the bias term. This represents the Sigmoid activation function.

[0075] In this embodiment, the spatial coordinate inversion point cloud generation module acquires a dish instance segmentation mask and a depth image, traverses all valid pixels in the dish instance segmentation mask, extracts the corresponding depth values, and for each valid pixel coordinate (u, v) in the dish instance segmentation mask, obtains the depth value from the depth image D. Generate a list of depth values Simultaneously, it records the mapping relationship between pixel coordinates and depth values; using camera intrinsic parameters, it converts the pixel coordinates (focal length: , Principal point coordinates: , The depth value is converted into three-dimensional spatial coordinates (X, Y, Z). The coordinate transformation logic is as follows:

[0076] Horizontal offset:

[0077] ;

[0078] Vertical offset:

[0079] ;

[0080] The depth value is directly used as the Z coordinate. ;

[0081] For each valid pixel, calculate (X, Y, Z) to generate an initial point cloud, and organize the 3D coordinates into structured point cloud data:

[0082] Create an independent point cloud list for each dish instance, containing all (X, Y, Z) coordinates;

[0083] Add dish identifiers and segmentation confidence to ensure accurate association between point clouds and dish instances;

[0084] Remove points with invalid depth values, such as Z≤0, retain valid point clouds, and improve the purity of point cloud data;

[0085] Generate a standardized initial 3D point cloud data package.

[0086] In this embodiment, the food geometric model reconstruction module acquires an initial three-dimensional point cloud, calculates the number of neighboring points for each point, removes outliers with fewer than a threshold number of neighboring points, eliminates sensor noise and isolated points, and, based on the point cloud density, removes points with fewer than a preset number of points within the radius to generate a purified point cloud.

[0087] Specifically, a planar model is fitted into the clean point cloud, and the bottom plane of the plate is determined by minimizing the distance error from the points to the plane. a, b, c represent the normal vectors of the plane. d represents a constant term related to the planar position, and x, y, z represent any point on the plane. Calculate the distance from each point to the fitted plane. Remove Points smaller than the preset value belong to the bottom of the plate and are retained. Points exceeding the preset value belong to the dish category.

[0088] Specifically, the dish point cloud is input into the Baisong reconstruction, and implicit functions are solved. Generate a smooth and continuous triangular mesh model, adjust the reconstruction depth and resolution, perform edge detection on the reconstructed model, remove discontinuous patches, and generate a closed, five-hole geometric model of the dish.

[0089] In this embodiment, the multi-strategy adaptive volume calculation module obtains the dish's geometric model, calculates the angle between the normal vector of all triangular facets of the dish's geometric model and the normal vector outside the model, and determines that the model is closed if all the angles are less than a preset angle; otherwise, it is determined to be an open model.

[0090] Specifically, the density threshold is adaptively adjusted based on the height of the dish, and the actual volume of the model is calculated using the triangular mesh volume integral formula, resulting in:

[0091] ,

[0092] In the formula, V represents the total volume of the entire geometric model of the dish. The coefficient representing the volume calculation. , , This represents the coordinate vectors of the three vertices of the triangle.

[0093] Preferably, the data output module acquires the dish type and volume results, generates a standardized data packet in a unified format, automatically selects the output method according to the system's preset configuration, verifies the validity of the API endpoint, authentication token, and queue connection status in real time, sends a standardized JSON data packet through the selected interface, and generates push logs.

[0094] In this embodiment, color and depth data acquired by a depth camera are used as input, and the construction and utilization of 3D point clouds of dishes are taken as the core link. The complete process of plate detection, dish recognition, and volume estimation is achieved through a single shot. In actual use, the system first continuously acquires images and depth information of the plate area. By analyzing the short-term changes in the edge contour and depth field of the plate, it determines whether the plate is stable, thus avoiding volume estimation errors caused by shaking. After a stable frame is detected, the system automatically completes plate detection and dish recognition based on the color image and some depth information using a deep learning model. It performs instance-level segmentation of different dishes on the plate to obtain pixel region masks corresponding to each type of dish. Next, the system no longer stops at the two-dimensional mask level, but combines the camera's intrinsic and extrinsic parameters to back-project the depth pixels in these masks point by point into three-dimensional space, constructing a 3D point cloud of dishes with real scale. Through filtering, noise removal, and plane fitting, the system separates the dish point cloud data that fits the actual geometric shape. Building upon this foundation, the volume estimation algorithm directly operates on the point cloud itself. It can employ methods such as height integration, voxel grid accumulation, or envelope approximation to solve for the volume of each dish's point cloud. Ultimately, the dish category, 3D point cloud volume, and related confidence information are output in a structured format, which can be integrated with payment systems, nutrition analysis modules, or data acquisition platforms. For the user, the entire process is simply placing the plate in the designated location; no additional operation is required, and the system automatically completes the entire processing chain from image acquisition to 3D volume output.

[0095] Working principle: The system controls the color camera and depth camera through the underlying driver interface of the depth camera, ensuring that the two image acquisitions are completely synchronized on the time axis, avoiding data misalignment caused by the movement of the plate; the system uses the intrinsic and extrinsic parameters obtained from camera calibration to accurately transform the pixel coordinate system of the depth image to the coordinate system of the color image, and establishes the depth-color pixel mapping relationship through bilinear interpolation; the depth image undergoes preliminary filtering to remove noise points based on the statistical characteristics of local depth values, and a region growing algorithm is used to repair depth holes caused by occlusion, generating a continuous depth distribution; finally, timestamp alignment is performed, and the data alignment effect is verified through visualization.

[0096] Based on the acquired color images, the system uses object detection algorithms to accurately locate the plate region in the images and outputs the plate's bounding box information. The system analyzes the depth value changes of consecutive frames within the bounding box, calculates the average depth value and statistical characteristics of the plate region, and obtains robust depth information through a depth gradient-guided weighted averaging method. Then, it tracks the average depth value changes of consecutive frames, calculates the normalized depth change, and compares this change with a preset threshold to determine whether the plate is in a stable state. Only when the plate is in a stable state does the system continue to trigger subsequent food recognition and volume calculation processes. The system receives stable frame color and depth images, performs resolution adjustment and spatial alignment verification, and then stitches the depth image as the fourth channel with the RGB image. The system generates a four-channel input tensor and calculates the gradient magnitude of the depth image to generate a depth-guided mask. This mask adaptively adjusts the fusion weights of RGB and depth features based on gradient changes, enhancing depth feature weights in the edge regions of the dish while preserving RGB features in the inner regions. Through the depth-guided feature fusion mechanism, the system performs end-to-end instance segmentation inference, generating a high-precision pixel-level dish segmentation mask. Using the mask generated by the dish instance segmentation module and the corresponding depth image, the system traverses all valid pixels in the mask, extracts the corresponding depth values, and records the mapping relationship between pixel coordinates and depth values. The system converts pixel coordinates and depth values ​​into three-dimensional spatial coordinates using camera intrinsics and calculates the horizontal and vertical offsets. The quantity and depth values ​​are used as Z-coordinates; 3D coordinates are calculated for each valid pixel to generate an initial point cloud; the system organizes the 3D coordinates into structured point cloud data, creates an independent point cloud list for each dish instance, adds dish identifiers and segmentation confidence scores, and removes invalid depth value points; upon receiving the initial 3D point cloud, point cloud filtering is first performed to remove outliers with insufficient neighborhood points and noise points in low-density areas, generating a purified point cloud; a planar model is fitted to the purified point cloud, and the bottom plane of the plate is determined by minimizing the distance error from the point to the plane. The distance from each point to the plane is calculated, and points belonging to the bottom of the plate are removed, retaining the point cloud belonging to the dish; the dish point cloud is input into the Poisson reconstruction algorithm, which generates a smooth and continuous 3D point cloud by solving implicit functions. The system uses a triangular mesh model, adjusting reconstruction parameters to ensure complete model details. Edge detection and patch optimization are performed on the reconstructed model to remove discontinuous patches, generating a closed and hole-free food geometry model. Based on the optimized food geometry model, the model's closure is first verified by calculating the angle between the triangular patch normals and the model's external normals to determine if the model is closed. The density threshold is adaptively adjusted according to the food height to determine the model's quality level. For high-quality closed models, the volume is calculated using triangular mesh volume integration; for low-quality or open models, voxelization integration is used. The system uses a cross-validation mechanism to ensure the reliability of the calculation results, comparing the model volume with the initial point cloud volume and dynamically optimizing the calculation strategy. Finally, the volume value is output.The system receives dish identification results and volume calculation data, standardizes and encapsulates them according to a unified data model, and generates a structured data packet containing dish ID, name, volume value, calculation strategy, and confidence level. Based on the system's preset configuration, it automatically selects the optimal output method, such as RESTful API, message queue, or database write, and verifies the API endpoint validity, authentication token, and queue connection status in real time. It then sends standardized JSON data packets through the selected interface, generating detailed push logs.

[0097] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their likenesses.

[0098] The present invention and its embodiments have been described above. This description is not restrictive, and the accompanying drawings are only one embodiment of the present invention; the actual structure is not limited thereto. In conclusion, if those skilled in the art are inspired by this description and design similar structures and embodiments without departing from the spirit of the invention, such designs should fall within the protection scope of the present invention.

Claims

1. An automatic plate food identification and volume estimation system based on RGB-D vision, characterized in that: It includes a multimodal data acquisition and preprocessing module, a plate perception stability judgment module, a dish instance segmentation module, a spatial coordinate inversion point cloud generation module, a dish geometric model reconstruction module, a multi-strategy adaptive volume calculation module, and a data output module; The multimodal data acquisition and preprocessing module acquires data through a depth camera, obtaining a color image and a depth image that are synchronously aligned with the plate area; The plate stability judgment module detects the plate area and analyzes short-term depth changes based on the acquired color image to determine the plate stability. The dish instance segmentation module segments dish instances based on the determined stable frames using fused RGB-D features, generating a pixel-level segmentation mask for each dish. The spatial coordinate inversion point cloud generation module performs three-dimensional spatial coordinate inversion based on the segmentation mask and the corresponding depth image to construct the initial three-dimensional point cloud for each dish. The food geometric model reconstruction module performs filtering, plate plane removal, and surface reconstruction based on the initial 3D point cloud to obtain an optimized food geometric model. The multi-strategy adaptive volume calculation module performs volume calculations based on the optimized dish geometry model; The data output module performs data standardization and encapsulation based on the dish category and volume results, and outputs the data through the API interface.

2. The automatic plate food identification and volume estimation system based on RGB-D vision according to claim 1, characterized in that: The multimodal data acquisition and preprocessing module sets the exposure time, frame rate, and trigger signal of the color camera and the depth camera through the underlying driver interface of the depth camera, enables hardware-level synchronization, transforms the pixel coordinate system of the depth image to the coordinate system of the color image through the intrinsic and extrinsic parameter matrices obtained from camera calibration, performs coordinate system transformation through rotation matrix R and translation vector T, calculates the corresponding position of each pixel in the depth image in the color image using bilinear interpolation, generates a depth-color pixel mapping table, removes outlier noise points based on the local statistical characteristics of depth values, fills holes in the depth image caused by occlusion using region growing, repairs missing data, generates a continuous depth distribution, and performs timestamp alignment processing on the acquired depth image and color image.

3. The automatic plate food identification and volume estimation system based on RGB-D vision according to claim 1, characterized in that: The plate-sensing stability judgment module acquires a color image, locates the plate area in the image through target detection, and outputs new coordinates within the plate's bounding box. This is achieved as follows: ， In the formula, , This indicates the center coordinates of the detected bounding box of the plate. , Indicates the center coordinates of the anchor frame. , This represents the bounding box regression offset. and These represent the width and height of the anchor frame, respectively. This indicates the ratio of the plate height to the constraint factor; Obtain the depth values ​​of all pixels within the bounding box of the plate. By combining depth sensor data, the plate area is detected and the corresponding depth value is generated, which is achieved as follows: ， In the formula, This represents the weighted average depth value of the plate area. This represents the depth value of the i-th pixel within the bounding box of the plate. This represents the mean value of the depth of the plate area. The standard deviation of the depth values ​​of the plate area is represented by N, which represents the total number of pixels within the plate's bounding box.

4. The automatic plate food identification and volume estimation system based on RGB-D vision according to claim 3, characterized in that: The plate sensing stability judgment module acquires the average depth value of consecutive frames, tracks the depth value change of the plate area, and calculates the short-term depth change, as follows: ， In the formula, This represents the short-term depth change. This represents the average depth value of the current frame. M represents the average depth value of the previous frame, and M represents the number of consecutive frames. Indicates the amount of depth change in consecutive frames; A preset stability threshold is set, and the short-term depth change is compared with the preset stability threshold. If the short-term depth change is lower than the threshold, the plate is determined to be in a stable state. Conversely, it is determined to be an unstable state.

5. The automatic plate food identification and volume estimation system based on RGB-D vision according to claim 1, characterized in that: The dish instance segmentation module acquires a stable frame color image and a depth image, concatenates the depth image as the fourth channel with the RGB image to generate a four-channel input tensor (R, G, B, D), calculates the depth gradient map, and generates a depth guiding mask. .

6. The automatic plate food identification and volume estimation system based on RGB-D vision according to claim 5, characterized in that: The dish instance segmentation module generates a dish force segmentation mask based on a depth-guided mask, as follows: ， In the formula, This represents a food instance segmentation mask. Representing the features of an RGB image, Features representing depth images Let represent the deep feature enhancement coefficient, W represent the weights of the segmentation network, and b represent the bias term. This represents the Sigmoid activation function.

7. The automatic plate food identification and volume estimation system based on RGB-D vision according to claim 1, characterized in that: The spatial coordinate inversion point cloud generation module acquires a dish instance segmentation mask and a depth image, traverses all valid pixels in the dish instance segmentation mask, extracts the corresponding depth values, and for each valid pixel coordinate (u, v) in the dish instance segmentation mask, obtains the depth value from the depth image D. Generate a list of depth values The pixel coordinates and depth values ​​are converted into three-dimensional spatial coordinates (X, Y, Z) using camera intrinsic parameters; (X, Y, Z) are calculated for each valid pixel to generate an initial point cloud, and the three-dimensional coordinates are organized into structured point cloud data; a standardized initial three-dimensional point cloud data package is generated.

8. The automatic plate food identification and volume estimation system based on RGB-D vision according to claim 1, characterized in that: The food geometric model reconstruction module acquires an initial 3D point cloud, calculates the number of neighboring points for each point, removes outliers with fewer than a threshold neighboring points, eliminates sensor noise and isolated points, and removes points with fewer than a preset number of points within the radius based on the point cloud density, generating a purified point cloud. A planar model is fitted into the purified point cloud, and the bottom plane of the plate is determined by minimizing the distance error from the point to the plane. The distance from each point to the fitted plane is calculated. The food point cloud is then input into the Baisong reconstruction module, which generates a smooth and continuous triangular mesh model by solving implicit functions. The reconstruction depth and resolution are adjusted, edge detection is performed on the reconstructed model, discontinuous patches are removed, and the food geometric model is generated.

9. The automatic plate food identification and volume estimation system based on RGB-D vision according to claim 1, characterized in that: The multi-strategy adaptive volume calculation module acquires the dish's geometric model, calculates the angles between the normal vectors of all triangular facets of the model and the external normal vectors of the model. If all angles are less than a preset angle, the model is determined to be closed; otherwise, it is determined to be an open model. The density threshold is adaptively adjusted based on the dish's height, and the actual volume of the model is calculated using the triangular mesh volume integral formula. ， In the formula, V represents the total volume of the entire geometric model of the dish. The coefficient representing the volume calculation. , , This represents the coordinate vectors of the three vertices of the triangle.

10. The automatic plate food identification and volume estimation system based on RGB-D vision according to claim 1, characterized in that: The data output module acquires the dish type and volume results, generates a standardized data packet in a unified format, automatically selects the output method according to the system's preset configuration, verifies the validity of the API endpoint, authentication token, and queue connection status in real time, sends a standardized JSON data packet through the selected interface, and generates push logs.

Images