Visual identification dish distribution system for unmanned intelligent kitchen

Abstract

The invention relates to the technical field of intelligent kitchens, and particularly discloses a visual identification dish distribution system for an unmanned intelligent kitchen, and the system comprises a dish image collection module which is used for collecting a plane image and a depth image of a dish; the dish visual identification module is used for extracting visual features of dishes based on the plane image and the depth image and constructing a feature vector library; the mechanical arm motion path planning module is used for determining the grabbing position of the mechanical arm according to the visual features and planning a motion path according to the grabbing position of the mechanical arm; and the quantitative meal distribution module is used for monitoring the weight of the dishes in real time and controlling the mechanical arm to stop serving the dishes, accurate meal distribution is achieved through the modes of multi-dimensional feature matching, accurate grabbing position calculation, quantitative meal distribution combining the weight and the dish distribution proportion and the like, multiple requirements can be met, and the user experience is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of intelligent kitchens, and in particular to a visual recognition and food distribution system for unmanned intelligent kitchens. Background Art

[0002] In crowded dining venues like universities and large factories, canteen ordering and serving have long relied on manual labor. This is especially true during peak meal times, when manual serving exposes numerous drawbacks. For one thing, efficiency is extremely low, leading to long queues and long wait times, severely impacting the dining experience and canteen operational efficiency. Furthermore, precisely controlling portion sizes poses a challenge. Standardized operation by different serving staff is difficult to achieve, resulting in inconsistent portion sizes and a failure to meet the demand for standardized service.

[0003] With the rapid development of science and technology, cutting-edge technologies such as machine learning, deep learning, and artificial intelligence have gradually penetrated into the catering industry. Robotic food-serving systems have emerged, bringing hope for solving the difficulties of traditional manual food-serving. However, most existing food-serving devices have obvious flaws. They usually rely on fixed programs to perform food-serving tasks, which requires extremely high precision in the placement and removal of food. Once the position of the food-serving bowl or the food-retrieving bowl deviates slightly, it is very easy to cause food to spill, which not only causes food waste, but also brings unnecessary cleaning troubles and health hazards to the scene. At the same time, due to the diversity of dishes in terms of ingredient types and cooking methods, existing devices find it difficult to accurately control the grabbing amount of different dishes, and cannot ensure the consistency and accuracy of each meal.

[0004] In view of this, there is an urgent need for an innovative technical solution that can overcome the above-mentioned problems, realize the full automation and intelligence of the food distribution process, improve the efficiency and accuracy of food distribution, reduce manual operations and potential errors, and improve the quality of catering services and the level of hygiene and safety. Summary of the Invention

[0005] The purpose of the present invention is to provide a visual recognition and food distribution system for an unmanned intelligent kitchen, a system and a computer device to solve the technical problems raised in the above-mentioned background technology.

[0006] To achieve the above object, the present invention provides the following technical solutions:

[0007] A visual recognition and food distribution system for an unmanned intelligent kitchen, comprising:

[0008] The dish image acquisition module is used to collect the plane image and depth image of the dish;

[0009] A dish visual recognition module, configured to extract visual features of dishes based on the planar image and the depth image, and construct a feature vector library;

[0010] A robot arm motion path planning module, configured to determine a gripping position of the robot arm according to the visual features, and to plan a motion path according to the gripping position of the robot arm;

[0011] The quantitative food distribution module is used to monitor the weight of dishes in real time and control the robotic arm to stop serving food.

[0012] Preferably, the step of extracting visual features of the dish based on the planar image and the depth image and constructing a feature vector library includes:

[0013] Preprocessing the planar image and the depth image;

[0014] Extracting feature information of multiple dishes from the plane image and the depth image;

[0015] Fusing the plurality of dish feature information extracted from the planar image and the depth image to obtain a fused feature vector, and performing dimensionality reduction processing on the fused feature vector;

[0016] A feature vector library is constructed based on the fused feature vector after dimensionality reduction processing, and the new fused feature vector is matched with the fused feature vector in the existing feature vector library. If the match is successful, the feature vector library is not updated. If the match is unsuccessful, the new fused feature vector after dimensionality reduction processing is added to the feature vector library and the feature vector library is updated.

[0017] Preferably, the step of matching the new fusion feature vector after dimensionality reduction processing with the fusion feature vector in the existing feature vector library includes:

[0018] Calculate the dot product of the new fused feature vector and each fused feature vector in the feature vector library. The calculation formula is:

[0019]

[0020] Where, represents the dot product, a i1 represents the i1th new fusion feature vector, b i2 represents the fused feature vector in the i2th feature vector library, and n represents the total number of all fused feature vectors;

[0021] Calculate the modulus of the new fused feature vector and each fused feature vector in the feature vector library respectively, wherein the modulus calculation formula of the new fused feature vector is:

[0022]

[0023] Where, Represents the modulus of the new fused feature vector, a irepresents the i1th new fused feature vector, and n represents the total number of all fused feature vectors;

[0024] The formula for calculating the modulus length of each fused feature vector in the feature vector library is:

[0025]

[0026] Where, Indicates the modulus length of each fused feature vector in the feature vector library, b i2 represents each fused feature vector in the i2th feature vector library, and n represents the total number of all fused feature vectors;

[0027] Obtaining similarity between a new fused feature vector and a fused feature vector in a feature vector library according to the dot product and the modulus;

[0028]

[0029] In the formula, X1 represents the similarity, represents the dot product, represents the modulus of the new fused feature vector, Represents the modulus length of each fused feature vector in the feature vector library;

[0030] If the similarity reaches a preset value, the new fused feature vector and the fused feature vector in the feature vector library are matched successfully; if the similarity does not reach the preset value, the new fused feature vector and the fused feature vector in the feature vector library are matched unsuccessfully.

[0031] Preferably, the step of determining the gripping position of the robotic arm according to the visual features comprises:

[0032] Obtaining spatial depth information and two-dimensional area information of the dish from the visual features;

[0033] Get camera parameters, including camera intrinsic parameter matrix;

[0034] The Transformer-based semantic segmentation model processes the two-dimensional region information, outputs a segmentation mask for the dish area, and then traverses each pixel in the segmentation mask to find the pixel points belonging to the dish area.

[0035] Get the pixel point in the spatial depth information that has the same coordinates as the pixel point selected in the segmentation mask, and get the depth value of the pixel point;

[0036] Calculate the plane coordinates corresponding to the pixel point in the camera coordinate system according to the depth value and the camera intrinsic parameter matrix;

[0037] Determine the three-dimensional coordinates of the pixel point in the camera coordinate system according to the plane coordinates and the depth value;

[0038] Convert the coordinates in the camera coordinate system to the robotic arm coordinate system to obtain the robotic arm grasping position.

[0039] Preferably, the step of planning a motion path according to the grasping position of the robotic arm includes:

[0040] Obtain environmental information within the robotic arm's workspace, including the location, shape, and size of obstacles;

[0041] constructing an environmental map based on the environmental information;

[0042] The current position of the robot arm is used as the starting point. According to the determined grasping position, the position and posture of the robot arm end effector when it reaches the grasping point is set as the target point.

[0043] The RRT algorithm is used to generate the optimal grab and place path.

[0044] Preferably, the step of using the RRT algorithm to generate the optimal grab and place path includes:

[0045] Create a tree with the robot's starting point as the root node;

[0046] Randomly sample a point in the workspace of the robot;

[0047] Find the closest node in the tree;

[0048] Extend a step from the node to the point to obtain a new node;

[0049] If the new node does not collide, add it to the tree and connect the new node and the node;

[0050] The steps of sampling, finding the nearest point, expanding, and connecting are repeated until the target point is included in the tree or the maximum number of iterations is reached;

[0051] When the tree contains a target point, backtrack from the target point to the starting point and extract a path from the starting point to the target point. This path is the optimal grasping and placing path of the robot arm.

[0052] Preferably, the grab and place path uses a path smoothing algorithm to remove redundant points and sharp corners in the path to make the path smoother.

[0053] Preferably, the step of monitoring the weight of the dish in real time and controlling the robotic arm to stop serving the dish further comprises:

[0054] Get real-time weight information of dishes;

[0055] Get the side dish ratio information in the dish feature information

[0056] According to the needs of meal distribution, obtain the meal distribution weight threshold;

[0057] Whether the meal weight threshold is reached is determined based on the side dish ratio information and the real-time weight information; if the meal weight threshold is reached, the robotic arm is controlled to stop serving the food.

[0058] The present application also provides a computer device, comprising a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of the above system when executing the computer program.

[0059] The present application also provides a computer-readable storage medium having a computer program stored thereon, which implements the steps of the above system when executed by a processor.

[0060] The beneficial effects of this application are as follows: the unmanned intelligent kitchen visual recognition food distribution system achieves accurate food distribution through multi-dimensional feature matching, precise grasping position calculation, and quantitative food distribution based on weight and side dish ratio; with the help of RRT algorithm, it quickly plans obstacle avoidance paths and smoothes them for efficient operation; adopts segmentation models suitable for a variety of dishes and path planning algorithms suitable for different environments to meet diverse needs; reduces costs due to effective obstacle avoidance, reduced food waste and modular design; improves user experience by ensuring food distribution quality, meeting special needs and creating a quiet environment; assists in system optimization by recording data and provides support for related research and policy making. These functions work together to make the system accurate, efficient, adaptable, cost-effective, user-friendly and of high data value. BRIEF DESCRIPTION OF THE DRAWINGS

[0061] Figure 1 This is a system diagram of an embodiment of the present application.

[0062] Figure 2 This is a schematic diagram of the internal structure of a computer device according to an embodiment of the present application.

[0063] The realization of the objectives, functional features and advantages of this application will be further explained in conjunction with embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0064] It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0065] Example 1

[0066] like Figure 1 As shown, the present application provides a visual recognition and food distribution system for an unmanned intelligent kitchen, comprising:

[0067] The dish image acquisition module is used to capture planar and depth images of dishes. A high-resolution optical camera is used to capture RGB images of dishes. These images contain visual information such as the color, shape, and texture of the dishes, which can be used for visual recognition of dishes. An Intel RealSense camera is used to capture depth images of dishes on the food display. These images contain depth information of the dishes, which can be used to measure the position and height of the dishes.

[0068] A dish visual recognition module is used to extract visual features of dishes based on the planar image and depth image and construct a feature vector library. Deep learning technology, such as the ResNet50 model in the convolutional neural network (CNN), is used to extract visual features of dishes, such as color, shape, texture, etc., encode the visual content of the image into feature vectors, and construct a feature vector library for dishes.

[0069] A robot arm motion path planning module, configured to determine a gripping position of the robot arm according to the visual features, and to plan a motion path according to the gripping position of the robot arm;

[0070] When a new dish is added, it is first matched with the existing feature vector library. If the match is successful, the existing crawling process is used. If the match fails, the features of the new dish need to be added to the feature vector library and the model is updated at the same time. The dish vector currently extracted by ResNet50 is embedded in a 2048-dimensional Euclidean vector space; based on the semantic segmentation model of the Transformer architecture, SegFormer is selected as the main segmentation model. SegFormer outputs multi-scale features through its hierarchical TransformerEncoder, which can effectively capture local details and global information, thereby accurately outputting the segmentation mask of the dish area. By combining the depth image with the segmentation mask, through spatial transformation and geometric calculation, the three-dimensional coordinates (X, Y, Z) of the dish are obtained to determine the grasping position of the robotic arm. When the amount of food changes, the dish point can be changed in real time.

[0071] The robotic arm uses AI vision to accurately segment and deeply inspect dishes, removes the user's selected dish from the dish counter, and places it on the user's tray. Using the path planning and control interface provided by the ROS (Robot Operating System) framework and the RRT (Rapidly-exploring Random Trees) algorithm, the robotic arm generates the optimal grasping and placement path, enabling it to find obstacle-avoiding paths in complex environments.

[0072] The quantitative food distribution module is used to monitor the weight of dishes in real time and control the robotic arm to stop serving dishes. By equipping the automatic food serving table with a weight sensor, a control system and a mechanical device, the weight sensor is used to monitor the weight of dishes in real time. When the preset weight is reached, the control system drives the mechanical device to stop serving dishes, thereby achieving quantitative food distribution.

[0073] The step of extracting visual features of the dish based on the planar image and the depth image and constructing a feature vector library includes:

[0074] Preprocessing is performed on the planar image and depth image to improve image quality. For RGB images, a denoising algorithm is used to remove noise generated during the shooting process, and image enhancement technology is used to enhance the image contrast and brightness to make the characteristics of the dish more distinct. Depth correction and hole filling are performed on the depth image to ensure the accuracy of depth information. In addition, the image can be cropped and scaled to unify the image size according to subsequent processing requirements.

[0075] Extract multiple dish feature information from the plane image and depth image; for plane image extraction, deep learning technology, such as the ResNet50 model in the convolutional neural network (CNN), is used to extract features from the RGB image. The convolution layer in the model can automatically learn the local and global features of the dish, such as the edge, texture, shape, and color. After multiple layers of convolution and pooling operations, the visual content of the image is encoded into a feature vector; for depth images, a model based on the Transformer architecture is used, such as the Transformer Encoder in the SegFormer model. Through its hierarchical structure, multi-scale features are output to capture the depth changes of the dish in the depth image, the spatial position and shape information of the object, and are also encoded as feature vectors;

[0076] The plurality of dish feature information extracted from the plane image and the depth image are fused to obtain a fused feature vector, and the fused feature vector is subjected to dimensionality reduction processing; the features extracted from the plane image and the depth image are fused, and the two feature vectors are sequentially connected into a new feature vector by splicing; or a weighted summation method is used to assign weights according to the importance of the plane image and depth image features, and then add them to obtain a fused feature vector. The dimension of the fused feature vector may be high and contain redundant information. The principal component analysis (PCA) dimensionality reduction algorithm is used to reduce the dimension of the feature vector while retaining key information as much as possible, thereby reducing the amount of calculation and improving the efficiency of subsequent processing;

[0077] A feature vector library is constructed based on the fused feature vector after dimensionality reduction processing, and the new fused feature vector is matched with the fused feature vector in the existing feature vector library. If the match is successful, the feature vector library is not updated. If the match is unsuccessful, the new fused feature vector after dimensionality reduction processing is added to the feature vector library and the feature vector library is updated. The fused feature vector after dimensionality reduction processing is stored in the feature vector library, and each dish corresponds to a feature vector. For the convenience of management and retrieval, a corresponding relationship between the dish category and the feature vector is established. This can be achieved through a database table structure, recording the association between the dish name, category and other information and the feature vector. When there is a new dish, the image of the new dish is first extracted according to the above steps to extract the fused feature vector, and then matched with the fused feature vector in the existing feature vector library. If the match is successful, it means that there are similar dishes in the feature vector library, and the existing crawling process can be used. If the match fails, the feature vector of the new dish is added to the feature vector library, and the model is updated to meet the recognition requirements of the new dish.

[0078] The step of determining the grasping position of the robotic arm according to the visual features comprises:

[0079] Obtaining spatial depth information and two-dimensional area information of the dish from the visual features;

[0080] Get camera parameters, including camera intrinsic parameter matrix;

[0081] The Transformer-based semantic segmentation model processes the two-dimensional region information, outputs a segmentation mask for the dish area, and then traverses each pixel in the segmentation mask to find the pixel points belonging to the dish area.

[0082] Get the pixel point in the spatial depth information that has the same coordinates as the pixel point selected in the segmentation mask, and get the depth value of the pixel point;

[0083] Calculate the plane coordinates corresponding to the pixel point in the camera coordinate system according to the depth value and the camera intrinsic parameter matrix;

[0084] Determine the three-dimensional coordinates of the pixel point in the camera coordinate system according to the plane coordinates and the depth value;

[0085] Convert the coordinates in the camera coordinate system to the robotic arm coordinate system to obtain the robotic arm grasping position.

[0086] In the visual recognition and food sorting system for unmanned smart kitchens, accurately determining the robotic arm's grasping position is crucial for achieving precise food sorting. This step determines the grasping position based on the acquired visual features of the dish. The specific process is as follows: First, the system obtains the dish's spatial depth information and two-dimensional area information from the visual feature data source. Spatial depth information reflects information such as the dish's position and distance in three-dimensional space, while the two-dimensional area information shows the distribution of the dish on the image plane. Simultaneously, the camera's intrinsic parameter matrix is ​​obtained. This matrix contains inherent camera parameters such as focal length and optical center position, which is an important basis for subsequent coordinate transformations. Next, a semantic segmentation model based on the Transformer architecture is used to process the two-dimensional area information. This model, through its unique self-attention mechanism, effectively captures long-range dependencies in the image, thereby accurately outputting a segmentation mask for the dish area. The segmentation mask is a binary image in which pixels in the dish area are labeled as foreground (usually 1) and background pixels are labeled as background (usually 0). The system then iterates through each pixel in the segmentation mask to select those belonging to the dish area. Next, the algorithm locates the pixel with the same coordinates as the pixel selected in the segmentation mask within the spatial depth information and obtains its depth value. This depth value represents the distance from the dish to the camera at that pixel. Based on this depth value and the camera's intrinsic parameter matrix, a specific geometric calculation method is used to calculate the corresponding plane coordinates of the pixel in the camera coordinate system. The plane coordinates and depth value are then combined to determine the three-dimensional coordinates of the pixel in the camera coordinate system. Finally, using a pre-calibrated transformation relationship, the coordinates in the camera coordinate system are converted to the robot coordinate system to determine the robot's grasping position. This position information is critical for the robot's accurate grasping operation, ensuring that it can precisely locate the appropriate grasping point on the dish. By comprehensively utilizing the dish's spatial depth information and two-dimensional area information, combined with advanced semantic segmentation models and precise coordinate transformation calculations, the robot's grasping position can be accurately determined, reducing grasping failures or deviations caused by inaccurate positioning, and improving the accuracy and reliability of the robot's grasping operation during the food distribution process. The semantic segmentation model based on the Transformer architecture can adapt to the complex shapes and texture features of different types of dishes and accurately segment the dish area. At the same time, the entire process of determining the grasping position can handle a variety of different camera parameters and spatial depth information, making the system highly adaptable to different shooting environments and camera settings, and can be widely used in a variety of unmanned smart kitchen scenarios. Accurate grasping position determination enables the robotic arm to perform grasping actions more quickly and efficiently, avoiding repeated operations or adjustments due to inaccurate positioning, thereby saving time in dish distribution, improving the efficiency of dish distribution in unmanned smart kitchens, and meeting the needs of large-scale, rapid dish distribution.Precise robotic gripping ensures uniform and stable food distribution, provides consistent food portions, and enhances the user's dining experience. It also reduces spillage or waste of food due to gripping issues, which is in line with environmental protection and conservation concepts. This method for determining the gripping position has clear steps and data processing flows, making it easy to integrate with other system modules of the unmanned smart kitchen (such as the visual acquisition module, robotic arm control module, etc.). Furthermore, when the camera needs to be replaced or the robotic arm layout needs to be adjusted, only the relevant parameter calibration and conversion calculations need to be re-performed to quickly adapt to the new system configuration, facilitating system expansion and upgrades.

[0087] The step of planning a motion path according to the grasping position of the robotic arm comprises:

[0088] Obtain environmental information within the robotic arm's workspace, including the location, shape, and size of obstacles;

[0089] constructing an environmental map based on the environmental information;

[0090] The current position of the robot arm is used as the starting point. According to the determined grasping position, the position and posture of the robot arm end effector when it reaches the grasping point is set as the target point.

[0091] The RRT algorithm is used to generate the optimal grab and place path.

[0092] In the visual recognition food distribution system of an unmanned smart kitchen, the robotic arm needs to move from its current position to the grasping position and complete the food distribution operation, so it is crucial to properly plan its movement path. The specific steps are as follows:

[0093] First, the system collects environmental information through various sensors (such as lidar, cameras, etc.) installed in the robot's workspace, covering detailed data such as the location, shape, and size of obstacles. This information is the basis for subsequent operations and directly affects the safety and feasibility of the robot's movement. Next, an environmental map is constructed based on the collected environmental information. This map presents the environmental conditions in the workspace in a digital form, providing an intuitive and accurate reference for the robot to plan the movement path. Then, the starting point and target point of the movement are determined. The current position of the robot is set as the starting point. Based on the previously determined grasping position, the position and posture of the robot's end effector when it reaches the grasping point are further determined and set as the target point. The target point not only contains spatial position information, but also involves the posture information of the robot's end effector, which is crucial for accurately grasping dishes. Finally, the rapid exploration random tree (RRT) algorithm is used to generate the optimal grasping and placement path. The RRT algorithm is a sampling-based path planning algorithm. It explores the space by randomly sampling points within the workspace and gradually constructing a random tree. Starting from the starting point, it continuously expands nodes in random directions until it reaches the target point or meets certain termination conditions, thereby finding a feasible path from the starting point to the target point. In the process of building the random tree, the algorithm automatically avoids obstacles to ensure that the generated path is safe and effective. By pre-acquiring environmental information within the robot's workspace and constructing an environmental map, the system can understand the distribution of obstacles in advance. When using the RRT algorithm to plan a path, these obstacles can be effectively avoided, preventing the robot from colliding with obstacles, ensuring the safety of the robot itself, as well as the safety of surrounding equipment and the environment, and reducing the risk of equipment damage and failure. Based on the characteristics of random sampling, the RRT algorithm can quickly explore and find feasible paths in complex workspaces. Compared to some traditional path planning algorithms, this method does not require an exhaustive search of the entire workspace, significantly reducing computational effort and planning time. This method meets the real-time requirements of the unmanned intelligent kitchen food-serving system, enabling the robotic arm to quickly respond and execute food-serving tasks. This path planning method is adaptable to workspaces of varying shapes and layouts, as well as complex obstacle distribution scenarios. The RRT algorithm generates appropriate motion paths for both confined spaces with dense obstacles and more open environments with irregularly shaped obstacles. This enables the unmanned intelligent kitchen food-serving system to operate stably in a variety of practical scenarios, enhancing the system's versatility and adaptability. Accurately planning the robotic arm's motion path from its current position to the grasping position helps the arm perform grasping and serving operations more efficiently, reducing unnecessary movement and time waste, and improving overall serving efficiency. Furthermore, stable and reliable path planning ensures the consistency and accuracy of serving actions, improving the quality and consistency of serving, and providing a better service experience for users.

[0094] The steps of using the RRT algorithm to generate the optimal grab and place path include:

[0095] Create a tree with the robot's starting point as the root node;

[0096] Randomly sample a point in the workspace of the robot;

[0097] Find the closest node in the tree;

[0098] Extend a step from the node to the point to obtain a new node;

[0099] If the new node does not collide, add it to the tree and connect the new node and the node;

[0100] The steps of sampling, finding the nearest point, expanding, and connecting are repeated until the target point is included in the tree or the maximum number of iterations is reached;

[0101] When the tree contains a target point, backtrack from the target point to the starting point and extract a path from the starting point to the target point. This path is the optimal grasping and placing path of the robot arm.

[0102] In an unmanned intelligent kitchen visual recognition food-sorting system, a Rapid Exploration Random Tree (RRT) algorithm is used to generate the optimal path for the robotic arm to safely and efficiently move from its starting position to the grasping position and complete the placement operation. The specific steps are as follows: First, a tree is created with the robotic arm's starting point as the root node. This tree serves as the foundation for subsequent exploration of the robotic arm's workspace, with the root node representing the robotic arm's initial position. Next, a point is randomly sampled within the robotic arm's workspace. The workspace is the area within which the robotic arm can move. The purpose of random sampling is to extensively explore possible paths within this space. Next, the node closest to the randomly sampled point is found in the created tree. This operation determines the starting point in the tree for expansion toward the sampled point. Next, a step size is added from the nearest node toward the randomly sampled point to obtain a new node. The step size is a pre-set distance for each expansion. This step-by-step expansion method explores the workspace. If the new node does not collide with obstacles within the workspace during expansion, it is added to the tree and connected to the nearest node. This step ensures that the tree expansion occurs within a safe and feasible area. The steps of sampling, finding the nearest point, expanding, and connecting are repeated until the tree contains the target point (i.e., the point corresponding to the position and posture of the robot's end-effector when it reaches the grasping point) or the pre-set maximum number of iterations is reached. This iterative process allows the tree to continuously grow and explore the workspace, gradually approaching the target point. When the tree contains the target point, it backtracks from the target point to the starting point to extract a path from the starting point to the target point. This path is the optimal grasping and placing path for the robot. The backtracking process follows the node connections in the tree to find a complete path from the starting point to the target, providing guidance for the robot's movement. The RRT algorithm uses random sampling to rapidly explore the robot's complex workspace. Compared to some traditional global search algorithms, it significantly reduces search time and improves path planning efficiency, meeting the fast response requirements of the robot arm in unmanned intelligent kitchen food distribution systems. During the node expansion process, the new node is added to the tree based on whether it collides with an obstacle. This ensures that the generated path always stays within the collision-free safe zone, effectively preventing collisions with obstacles in the workspace during movement and ensuring the safety of the robot arm and surrounding equipment. The algorithm is highly adaptable to the shape of the workspace and the distribution of obstacles. It can find a feasible path in both simple and complex work environments through random sampling and tree expansion. This enables the unmanned intelligent kitchen food distribution system to operate stably in various kitchen layouts and environmental conditions. As long as the target point is within the robot's workspace and a feasible path exists, the RRT algorithm will theoretically find a path from the starting point to the target point as the number of iterations increases.In practical applications, even in complex environments, a feasible path that meets the requirements can be found quickly to ensure that the robotic arm can complete the grasping and placement tasks.

[0103] The grab and place path uses a path smoothing algorithm to remove redundant points and sharp corners in the path, making the path smoother.

[0104] In an unmanned intelligent kitchen vision-based food distribution system, the robotic arm's grasping and placing path, generated using methods such as the RRT algorithm, is a feasible path from the starting point to the target point. However, this path may present some issues. For example, the path may contain redundant points, which are not necessary for the robotic arm to reach the target point from the starting point, but increase the complexity of path planning and execution. Furthermore, the path may contain sharp corners, requiring the robotic arm to frequently change direction and speed when navigating these corners. This places high demands on the robotic arm's mechanical structure and motion control, and may also lead to unstable motion. To address these issues, a path smoothing algorithm is used to process the grasping and placing path. The path smoothing algorithm analyzes the original path and identifies redundant points and sharp corners. The algorithm removes redundant points based on specific rules, simplifying the path. For sharp corners, the algorithm adjusts the path shape to smooth the corners, resulting in a smoother path. The smoothed path provides better guidance for the robotic arm's movements.

[0105] The step of monitoring the weight of the dish in real time and controlling the robotic arm to stop serving the dish further includes:

[0106] Get real-time weight information of dishes;

[0107] Get the side dish ratio information in the dish feature information

[0108] According to the meal distribution requirements, obtain the meal distribution weight threshold;

[0109] Whether the meal weight threshold is reached is determined based on the side dish ratio information and the real-time weight information; if the meal weight threshold is reached, the robotic arm is controlled to stop serving the food.

[0110] In the visual recognition food distribution system for unmanned intelligent kitchens, quantitative food distribution is a key component. Real-time monitoring of food weight and controlling the robotic arm to stop serving are essential steps in achieving quantitative food distribution. The specific operational process is as follows: First, the system uses a high-precision weight sensor installed at the bottom of the serving container or other suitable location to obtain real-time food weight information. This information is continuously transmitted to the control system at a certain frequency, ensuring that the system can promptly monitor dynamic changes in food weight. Next, the system obtains the side dish ratio information from the dish characteristics. Side dish ratio information describes the proportion of various ingredients in a dish, such as the proportion of chicken, peanuts, and cucumber in a serving of Kung Pao Chicken. This information is crucial for accurately determining whether food distribution meets the requirements. Then, based on food distribution requirements, the control system pre-sets or obtains in real time a food distribution weight threshold. The food distribution weight threshold refers to the expected weight of each dish, for example, setting the main course weight threshold for an adult lunch at 200 grams. Finally, the control system determines whether the food distribution weight threshold has been met based on the side dish ratio information and real-time weight information. Specifically, the system analyzes the approximate weight of each ingredient in the real-time weight calculation based on the side dish ratio information to comprehensively determine whether the current dish's weight meets the required weight threshold. If the weight threshold is reached, the control system sends a stop-serving command to the robotic arm, which immediately stops upon receiving the command, completing the serving operation. Combining the real-time weight of the dish and the side dish ratio information to determine whether the serving weight threshold has been reached allows for more precise control of the weight of each portion. This system considers not only the total weight of the dish but also the proportions of the various ingredients, avoiding serving errors caused by uneven distribution of ingredients. It ensures that the weight and nutritional content of each serving meet the set standards, enabling precise and quantitative serving. By flexibly accessing the serving weight threshold and side dish ratio information, the system can adapt to diverse serving requirements. Whether serving food to different age groups, dietary preferences, or specific dish requirements, these requirements can be met by adjusting the corresponding parameters, enhancing the versatility and adaptability of the unmanned intelligent kitchen serving system. Precise serving control reduces food waste caused by inaccurate serving. This prevents dishes from being overweight or underweight, ensuring that each dish utilizes the right ingredients. This reduces costs, improves ingredient utilization, and aligns with environmental protection and resource conservation. Stable and accurate meal distribution provides users with consistent dish quality, avoiding dissatisfaction caused by uneven distribution. For users with special dietary needs, such as diabetics who strictly control the weight of staple foods, the system can meet these requirements through precise meal distribution, thereby enhancing the user's dining experience and satisfaction.

[0111] Example 2

[0112] The second embodiment is an improvement on the first embodiment, and the step of matching the new fusion feature vector after dimensionality reduction processing with the fusion feature vector in the existing feature vector library includes:

[0113] The dot product of the fused feature vector and each fused feature vector in the feature vector library is calculated as follows:

[0114]

[0115] Where, represents the dot product, a i1 represents the i1th new fusion feature vector, b i2 represents the fused feature vector in the i2th feature vector library, and n represents the total number of all fused feature vectors;

[0116] Calculate the modulus of the new fused feature vector and each fused feature vector in the feature vector library respectively, wherein the modulus calculation formula of the new fused feature vector is:

[0117]

[0118] Where, Represents the modulus of the new fused feature vector, a i represents the i1th new fused feature vector, and n represents the total number of all fused feature vectors;

[0119] The formula for calculating the modulus length of each fused feature vector in the feature vector library is:

[0120]

[0121] Where, Indicates the modulus length of each fused feature vector in the feature vector library, b i2 represents each fused feature vector in the i2th feature vector library, and n represents the total number of all fused feature vectors;

[0122] Obtaining similarity between a new fused feature vector and a fused feature vector in a feature vector library according to the dot product and the modulus;

[0123]

[0124] In the formula, X1 represents the similarity, represents the dot product, represents the modulus of the new fused feature vector, Represents the modulus length of each fused feature vector in the feature vector library;

[0125] If the similarity reaches a preset value, the new fused feature vector and the fused feature vector in the feature vector library are matched successfully; if the similarity does not reach the preset value, the new fused feature vector and the fused feature vector in the feature vector library are matched unsuccessfully.

[0126] In the unmanned intelligent kitchen visual recognition and food ordering system, in order to accurately identify dishes, it is necessary to match the newly acquired and dimensionally reduced fused feature vector with the fused feature vectors in the existing feature vector library. The matching method used in this step calculates similarity based on the dot product and modulus of the vectors. The dot product of the vectors reflects the degree of consistency in the direction of the two vectors, while the modulus represents the length of the vectors. By dividing the dot product by the product of the modulus of the two vectors, the resulting similarity value can measure the degree of similarity between the new fused feature vector and each vector in the feature vector library. When the similarity reaches the preset value, the two can be considered to be matched successfully, that is, the new dish is similar to a certain type of dish already in the feature vector library; if the preset value is not reached, the match is unsuccessful, indicating that the new dish may be a type that is not included in the feature vector library; by calculating the similarity based on vector dot product and modulus, the similarity between the new feature vector and the existing feature vector can be measured from multiple dimensions. Compared with a single feature matching method, it takes into account the comprehensive characteristics of the dish more comprehensively, thereby improving the accuracy of dish recognition and reducing misjudgments and missed judgments. In addition, this matching method can flexibly handle feature vectors of different dimensions and different modal fusions, and adapt to the representation of various dish features. Whether it is a dish with a simple shape or a dish with a complex texture and structure, it can be accurately identified through the matching of feature vectors, which enhances the adaptability of the unmanned intelligent kitchen visual recognition and food distribution system to different dishes. In the specific implementation process, operations such as the calculation of dot product and module length and the comparison of similarity can be efficiently implemented through computer programs. The calculation process is relatively simple and does not require complex model training and parameter adjustment. It can complete the matching of a large number of feature vectors in a relatively short time, thereby improving the system's operating efficiency and meeting the needs of real-time food distribution in unmanned intelligent kitchens. When the system needs to add a new type of dish, it only needs to match the fused feature vector of the new dish with the existing feature vector library according to the above matching steps. If the match is unsuccessful, it can be included in the feature vector library to realize automatic updating and expansion of the system, which can adapt to the recognition needs of new dishes conveniently and quickly, and reduce the cost of system maintenance and upgrades.

[0127] like Figure 2 As shown, the present invention also provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor implements the operation of the above system when executing the computer program.

[0128] The present invention also provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the operation of the above system is realized.

[0129] Those skilled in the art will appreciate that all or part of the processes in the above-mentioned embodiment methods can be implemented by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media provided in this application and used in the embodiments may include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (SSRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct RAMbus dynamic RAM (DRDRAM), and RAMbus dynamic RAM (RDRAM).

[0130] It should be noted that, in this document, the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, apparatus, article, or method that includes a series of elements includes not only those elements but also other elements not explicitly listed, or includes elements inherent to such process, apparatus, article, or method. In the absence of further limitations, an element defined by the phrase "comprises a ..." does not exclude the presence of other identical elements in the process, apparatus, article, or method that includes the element.

[0131] The above description is only a preferred embodiment of the present invention and does not limit the patent scope of the present invention. Any equivalent structure or equivalent process transformation made by using the contents of the present invention description and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present invention.

Claims

1. A visual recognition and food distribution system for unmanned intelligent kitchens, characterized by: include: The dish image acquisition module is used to collect the plane image and depth image of the dish; A dish visual recognition module, configured to extract visual features of the dish based on the planar image and the depth image, and construct a feature vector library; A robot arm motion path planning module, configured to determine a gripping position of the robot arm according to the visual features, and to plan a motion path according to the gripping position of the robot arm; The quantitative food distribution module is used to monitor the weight of dishes in real time and control the robotic arm to stop serving food.

2. The visual recognition and food distribution system for an unmanned intelligent kitchen according to claim 1 is characterized in that: The step of extracting visual features of the dish based on the planar image and the depth image and constructing a feature vector library includes: Preprocessing the planar image and the depth image; Extracting feature information of multiple dishes from the plane image and the depth image; Fusing the plurality of dish feature information extracted from the planar image and the depth image to obtain a fused feature vector, and performing dimensionality reduction processing on the fused feature vector; A feature vector library is constructed based on the fused feature vector after dimensionality reduction processing, and the new fused feature vector is matched with the fused feature vector in the existing feature vector library. If the match is successful, the feature vector library is not updated. If the match is unsuccessful, the new fused feature vector after dimensionality reduction processing is added to the feature vector library and the feature vector library is updated.

3. The visual recognition and food distribution system for an unmanned intelligent kitchen according to claim 1 is characterized in that: The step of matching the new fusion feature vector after dimensionality reduction processing with the fusion feature vector in the existing feature vector library includes: Calculate the dot product of the new fused feature vector and each fused feature vector in the feature vector library. The calculation formula is: Where, represents the dot product, a i1 represents the i1th new fusion feature vector, b i2 represents the fused feature vector in the i2th feature vector library, and n represents the total number of all fused feature vectors; Calculate the modulus of the new fused feature vector and each fused feature vector in the feature vector library respectively, wherein the modulus calculation formula of the new fused feature vector is: Where, Represents the modulus of the new fused feature vector, a i represents the i1th new fused feature vector, and n represents the total number of all fused feature vectors; The formula for calculating the modulus length of each fused feature vector in the feature vector library is: Where, Indicates the modulus length of each fused feature vector in the feature vector library, b i2 represents each fused feature vector in the i2th feature vector library, and n represents the total number of all fused feature vectors; Obtaining similarity between a new fused feature vector and a fused feature vector in a feature vector library according to the dot product and the modulus; In the formula, X1 represents the similarity, represents the dot product, represents the modulus of the new fused feature vector, Represents the modulus length of each fused feature vector in the feature vector library; If the similarity reaches a preset value, the new fused feature vector and the fused feature vector in the feature vector library are matched successfully; if the similarity does not reach the preset value, the new fused feature vector and the fused feature vector in the feature vector library are matched unsuccessfully.

4. The visual recognition and food distribution system for an unmanned intelligent kitchen according to claim 1 is characterized in that: The step of determining the grasping position of the robotic arm according to the visual features comprises: Obtaining spatial depth information and two-dimensional area information of the dish from the visual features; Get camera parameters, including camera intrinsic parameter matrix; The Transformer-based semantic segmentation model processes the two-dimensional region information, outputs a segmentation mask for the dish area, and then traverses each pixel in the segmentation mask to find the pixel points belonging to the dish area. Get the pixel point in the spatial depth information that has the same coordinates as the pixel point selected in the segmentation mask, and get the depth value of the pixel point; Calculate the plane coordinates corresponding to the pixel point in the camera coordinate system according to the depth value and the camera intrinsic parameter matrix; Determine the three-dimensional coordinates of the pixel point in the camera coordinate system according to the plane coordinates and the depth value; Convert the coordinates in the camera coordinate system to the robotic arm coordinate system to obtain the robotic arm grasping position.

5. The visual recognition and food distribution system for an unmanned intelligent kitchen according to claim 1 is characterized in that: The step of planning a motion path according to the grasping position of the robotic arm comprises: Obtain environmental information within the robotic arm's workspace, including the location, shape, and size of obstacles; constructing an environmental map based on the environmental information; The current position of the robot arm is used as the starting point. According to the determined grasping position, the position and posture of the robot arm end effector when it reaches the grasping point is set as the target point. The RRT algorithm is used to generate the optimal grab and place path.

6. The visual recognition and food distribution system for an unmanned intelligent kitchen according to claim 1 is characterized in that: The steps of using the RRT algorithm to generate the optimal grab and place path include: Create a tree with the robot's starting point as the root node; Randomly sample a point in the workspace of the robot; Find the closest node in the tree; Extend a step from the node to the point to obtain a new node; If the new node does not collide, add it to the tree and connect the new node and the node; The steps of sampling, finding the nearest point, expanding, and connecting are repeated until the target point is included in the tree or the maximum number of iterations is reached; When the tree contains a target point, backtrack from the target point to the starting point and extract a path from the starting point to the target point. This path is the optimal grasping and placing path of the robot arm.

7. The visual recognition and food distribution system for an unmanned intelligent kitchen according to claim 1 is characterized in that: The grab and place path uses a path smoothing algorithm to remove redundant points and sharp corners in the path, making the path smoother.

8. The visual recognition and food distribution system for an unmanned intelligent kitchen according to claim 1 is characterized in that: The step of monitoring the weight of the dish in real time and controlling the robotic arm to stop serving the dish further includes: Get real-time weight information of dishes; Get the side dish ratio information in the dish feature information According to the needs of meal distribution, obtain the meal distribution weight threshold; Whether the meal weight threshold is reached is determined based on the side dish ratio information and the real-time weight information; if the meal weight threshold is reached, the robotic arm is controlled to stop serving the food.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, and the processor implements the above-mentioned visual recognition and food distribution system for an unmanned intelligent kitchen when executing the computer program.

10. A computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the above-mentioned visual recognition and food distribution system for an unmanned intelligent kitchen.

Images