Mouth identifying and positioning method and device and feeding mechanical arm

Abstract

The invention discloses a mouth recognition and positioning method in the technical field of mechanical arms, and the method comprises the steps: collecting a first image and a second image within a certain distance, the first image being the opening and closing state of the mouth of a human face, and the second image being a food type data set; and performing target position and shape labeling on the first image and the second image, and converting the labeled positions and shapes into a data format which can be trained by a model to form a first data set and a second data set. The feeding mechanical arm with the visual real-time positioning system is formed in the mode that an embedded model and the binocular camera are combined, the real-time performance and accuracy of recognition can be considered by building the embedded model, the binocular camera is selected to obtain three-dimensional information of the mouth of the human face, the distance measuring error can be reduced, the positioning accuracy can be improved, and the feeding mechanical arm has the advantages of being simple in structure and convenient to use. A temperature sensor and a heating sensor are embedded in the mechanical arm base, so that food can be kept in a constant-temperature state conveniently, and the dining experience of a user is improved.

Description

Technical Field

[0001] This invention relates to the field of robotic arm technology, specifically a mouth recognition and positioning method, device, and feeding robotic arm. Background Technology

[0002] People with limited upper limb movement and difficulty eating due to various reasons, especially the elderly and disabled, are highly dependent on others for assistance in their daily lives. During the eating process, this not only increases the care burden on families and medical institutions, but also reduces the patient's quality of life and autonomy.

[0003] To address the aforementioned issues, intelligent assisted mouth recognition and positioning methods, devices, and feeding robotic arms serve as innovative assistive tools, providing the possibility of autonomous feeding for people with mobility impairments.

[0004] However, existing mouth recognition and positioning methods, devices, and feeding robotic arms rely primarily on input modes such as buttons, levers, voice recognition, and head posture. This makes it difficult to adjust the feeding angle in real time when faced with changes in user posture, accidental collisions, or complex dining environments, resulting in a lack of flexibility and adaptability. Furthermore, traditional machine learning-based visual recognition technologies struggle to achieve a balance between real-time performance and accuracy, particularly in complex food environments and user mouth positioning, exhibiting issues such as inaccurate recognition, low positioning precision, and slow response. Additionally, current mouth recognition and positioning methods, devices, and feeding robotic arms mostly only provide food carrying capacity and lack constant food temperature control, failing to meet the temperature requirements of users with longer meal times or slower swallowing speeds. Summary of the Invention

[0005] The purpose of this invention is to provide a mouth recognition and positioning method, device, and feeding robotic arm. By combining an embedded model and a binocular camera, a feeding robotic arm with a real-time visual positioning system is constructed. By constructing an embedded model, both real-time recognition and accuracy can be taken into account. By using a binocular camera to obtain three-dimensional information of the human face and mouth, ranging errors can be reduced and positioning accuracy can be improved. By embedding temperature and heating sensors on the base of the robotic arm, it is convenient to keep the food at a constant temperature and improve the user's dining experience.

[0006] To achieve the above objectives, the present invention provides the following technical solution:

[0007] Firstly, a mouth recognition and localization method is provided, including:

[0008] Collect a first image and a second image within a certain distance. The first image is the opening and closing state of a human face's mouth, and the second image is a food type dataset.

[0009] The target location and shape are labeled in the first image and the second image respectively, and the labeled location and shape are converted into a data format that can be trained by the model to form the first dataset and the second dataset. The first dataset is the location dataset that can be trained by the model, and the second dataset is the shape dataset that can be trained by the model.

[0010] The first and second datasets are input into three iterative models respectively. Each iterative model extracts the target dataset. The first target model is obtained based on the difference between the target dataset and the expected dataset. The iterative model is an object detection and image segmentation model.

[0011] The classification text of the second image is obtained and an embedded model is constructed. The classification text is collected using the embedded model to obtain the second target model. The second target model is an embedded model selected based on the acquisition speed. The second target model is an integrated model of the binocular camera.

[0012] A stereo matching algorithm is used to obtain the disparity depth map of the first dataset. The disparity depth map is then input into the second target model, and the three-dimensional coordinates of the first dataset are output to the motion component. The motion component reaches the target position based on the three-dimensional coordinates.

[0013] Preferably, the data format for training the model includes a TXT document containing location coordinates and lip shape type information.

[0014] Preferably, the three iterative models are the fifth, eighth, and tenth iterations of the object detection and image segmentation model, wherein the object detection and image segmentation model is the YOLO model. By using iterative object detection and image segmentation models to process image data, the inference speed of the model can be maximized while ensuring the accuracy of recognition, and the amount of computation can be reduced. The training parameters are adjusted according to the training results of the fifth, eighth, and tenth iterations of the object detection and image segmentation model, which can quickly select the first target model. The tool for annotating the target position and shape and converting the annotated position and shape into a data format that the model can train is an image annotation tool, namely the LabelImg tool.

[0015] Preferably, the size of the first target model does not exceed 640 pixels and the average precision value is not less than 37.3. The first target model is a YOLOv8n model.

[0016] Preferably, the moving part is a robotic arm.

[0017] Preferably, obtaining the classification text of the second image includes:

[0018] A visual encoder is used to extract image features from the second image. The image features are then compared with preset text features using cosine similarity calculation. The similarity is calculated, and the preset text feature corresponding to the maximum similarity is output. The preset text feature corresponding to the maximum similarity is the classification text of the second image.

[0019] Preferably, the construction of the embedded model includes:

[0020] The target dataset is embedded into the initialization layer of the first target model using a multimodal pre-trained model to obtain an embedded model. The multimodal pre-trained model is a Clip model, and the embedded model is a YOLOv8n-Clip model.

[0021] Calculate the weights of the target dataset obtained by the embedded model.

[0022] Preferably, the calculation of the weights of the target dataset obtained by the embedded model includes:

[0023] The weighting calculation formula is as follows:

[0024] c = c box +c i

[0025] Calculate the weights of the target dataset;

[0026] Among them, c box c represents the bounding box of the target feature region. i This represents the entire image. `c` is the embedding vector extracted by the Clip model. Constructing an embedded model allows for rapid focusing on semantically relevant features. Calculating the weights of the target dataset improves the performance of the embedded model in handling small datasets and reduces annotation costs.

[0027] Secondly, a mouth recognition and positioning device is provided, including an embedded processor, a binocular camera, and a memory;

[0028] The memory is configured to store images and executable instructions;

[0029] The binocular camera is configured to acquire a first image and a second image within a certain distance, and send the acquired images to the memory for storage.

[0030] The embedded processor is configured to implement the mouth recognition and positioning method as described above when executing the executable instructions stored in the memory.

[0031] Thirdly, a feeding robotic arm is provided, including a base, a robotic arm, and a mouth recognition and positioning device as described above.

[0032] The embedded processor is installed on one side of the base, and a power switch is connected to one side of the embedded processor. The power switch is electrically connected to the power source. The binocular camera is fixed to the top of the robotic arm, and a feeder is axially connected to one end of the robotic arm.

[0033] A microcontroller is installed at the bottom of the base, integrating a PID temperature control system. A display screen is mounted on one side of the microcontroller, and a heating pad is placed on top of the microcontroller. A DS18B20 temperature sensor is connected to the top of the heating pad. The display screen is a touchscreen used to set the temperature threshold for the food on the plate and to display the plate temperature in real time. The microcontroller receives temperature data signals, converts them into digital signals, and sends them to the display screen. It also outputs signals to the heating pad through the PID temperature control system, which displays the temperature readings. The DS18B20 temperature sensor collects temperature data signals from the plate and sends them to the microcontroller. An embedded processor controls a binocular camera to recognize and locate the mouth of a person, and controls a robotic arm to deliver the feeder to the diner's mouth based on three-dimensional coordinates. Once the robotic arm delivers the feeder to the diner's mouth, the recognition and location of the person's mouth are complete, and feeding is achieved.

[0034] Preferably, the DS18B20 temperature sensor 4, heating pad 5, PID temperature control system, microcontroller 3, and power supply constitute an automated heating device. For heating food on plates, an integrated device based on the DS18B20 temperature sensor and PID temperature control system is used. Compared to the traditional DS18B20 temperature sensor, using the DS18B20 temperature sensor to obtain temperature data from the bottom of the plate eliminates the need for an additional analog-to-digital converter. Data is read by communicating with the microcontroller via digital signals, reducing the complexity of the system design. By adding a heating pad to the bottom of the plate, the PID (proportional-integral-derivative) temperature control system can heat the food according to the set temperature threshold, resulting in uniform and rapid heating. It can accurately control and automatically adjust the temperature of the food on the plate in real time, ensuring that the food is served within a suitable temperature range.

[0035] The DS18B20 temperature sensor 4 includes a thermistor and an analog-to-digital converter.

[0036] Preferably, the DS18B20 temperature sensor 4 has a temperature testing range of -55°C to +125°C. When the DS18B20 temperature sensor 4 is activated, it measures the temperature of the food on the plate and directly converts the temperature into a digital signal. Then, utilizing the 1-Wire bus protocol's single-wire communication capability, a series of pulses are transmitted to the microcontroller for parsing, converting the signal into an actual temperature value. Based on the optimal temperature range of 35°C to 45°C for dining, it determines whether the food needs heating and sets a temperature control threshold as the optimal dining temperature. If the PID temperature control system detects that the food temperature is below the minimum threshold value, it activates the heating pad 5. The PID temperature control system adjusts the system output through three independent control strategies: proportional (P), integral (I), and derivative (D). The output formula of the PID temperature control system is:

[0037]

[0038] In PID, "P" represents proportional control, which adjusts the control quantity based on the difference between the target value and the actual value; "I" represents integral control, which adjusts based on the accumulation of error over time, and its output is proportional to the integral of the error; and "D" represents derivative control.

[0039] Among them, K p K represents the proportional gain, e(t) represents the current error, and the larger the error, the stronger the system's adjustment capability. i It is the integral gain; the integral term can eliminate persistent steady-state errors, K d As a differential gain, the differential term can reduce system oscillation and overshoot. By manually controlling the three gain values, the temperature can be kept near the target temperature.

[0040] Compared with the prior art, the beneficial effects of the present invention are:

[0041] 1. In this invention, a feeding robotic arm with a real-time visual positioning system is constructed by combining an embedded model and a binocular camera. By constructing an embedded model, the real-time performance and accuracy of recognition can be balanced. The use of a binocular camera to acquire three-dimensional information of the human face and mouth can reduce ranging errors and improve positioning accuracy. By embedding temperature and heating sensors on the base of the robotic arm, the food can be kept at a constant temperature, improving the user's dining experience.

[0042] 2. This invention utilizes the multimodal text capabilities of the Clip algorithm to infer food categories not seen by the binocular camera through natural language without retraining. The model can be trained using a small dataset, reducing data collection costs. When the robotic arm is feeding the food, if the plate contains food that the user does not want, the Clip model can post-process the detection results of the YOLO model and identify the corresponding food based on the language description. This not only enables the rational allocation of computing resources but also enhances the accuracy of target retrieval and localization.

[0043] 3. This invention addresses food heating on plates by integrating a DS18B20 temperature sensor with a PID temperature control system. Compared to traditional DS18B20 temperature sensors, this invention utilizes the DS18B20 to acquire temperature data from the bottom of the plate, eliminating the need for an additional analog-to-digital converter. Data is read via digital signal communication with a microcontroller, reducing system design complexity. By adding a heating pad to the bottom of the plate, it works in conjunction with a PID (Proportional-Integral-Derivative) temperature control system to heat food according to a set temperature threshold, resulting in uniform and rapid heating. It allows for real-time, precise control and automatic adjustment of food temperature, ensuring the food is served within a suitable temperature range. Attached Figure Description

[0044] Figure 1 This is a flowchart illustrating the mouth recognition and positioning method of the present invention.

[0045] Figure 2 This is a structural diagram of the mouth recognition and positioning device of the present invention;

[0046] Figure 3 This is a structural diagram of the robotic arm of the present invention;

[0047] Figure 4 This is a flowchart of the binocular camera recognition and positioning process of the present invention;

[0048] Figure 5 This is a schematic diagram of the Clip model architecture of the present invention;

[0049] Figure 6 This is a diagram of the DS18B20 temperature sensor of the present invention;

[0050] Figure 7 This is a schematic diagram of the PID control system of the present invention;

[0051] Figure 8 This is a flowchart of the automated heating device of the present invention.

[0052] Figure 9 This is a training result diagram of the YOLO model for mouth recognition in this invention.

[0053] In the diagram: 1. Power switch; 2. Display screen; 3. Microcontroller; 4. DS18B20 temperature sensor; 5. Heating pad; 6. Embedded processor; 7. Robotic arm; 8. Binocular camera; 9. Feeder. Detailed Implementation

[0054] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0055] Although the steps in this invention are arranged by reference numerals, this is not intended to limit the order of the steps. Unless the order of the steps is explicitly stated or the execution of a step requires other steps as a basis, the relative order of the steps can be adjusted. It is understood that the term "and / or" as used herein refers to and covers any and all possible combinations of one or more of the associated listed items.

[0056] Please see Figure 1 This embodiment provides a mouth recognition and localization method, including the following steps:

[0057] S1: Collect the first and second images within a certain distance. The first image is the opening and closing state of a human face's mouth, and the second image is a food type dataset.

[0058] S2: Label the target location and shape of the first image and the second image respectively, and convert the labeled location and shape into a data format that the model can train, forming the first dataset and the second dataset. The first dataset is the location dataset that the model can train, and the second dataset is the shape dataset that the model can train.

[0059] S3: Input the first dataset and the second dataset into the three iterative models respectively. Each iterative model extracts the target dataset. The first target model is obtained based on the difference between the target dataset and the expected dataset. The iterative model is an object detection and image segmentation model.

[0060] S4: Obtain the classified text of the second image and construct an embedded model. Use the embedded model to collect the classified text to obtain the second target model. The second target model is an embedded model selected based on the acquisition speed. The second target model is an integrated model of the binocular camera.

[0061] S5: Use a stereo matching algorithm to obtain the disparity depth map of the first dataset, input the disparity depth map into the second target model, and output the three-dimensional coordinates of the first dataset to the motion component. The motion component reaches the target position according to the three-dimensional coordinates.

[0062] Preferably, the second target model is the fastest embedded model among multiple collected and classified texts.

[0063] Preferably, the three iterative models are the fifth, eighth, and tenth iterations of the object detection and image segmentation model. The object detection and image segmentation model is a YOLO model; the fifth iteration is a YOLOv5 model, the eighth iteration is a YOLOv8 model, and the tenth iteration is a YOLOv10 model. By using iterative object detection and image segmentation models to process image data, the inference speed of the model can be maximized while ensuring recognition accuracy, and the computational load can be reduced. Adjusting the training parameters based on the training results of the fifth, eighth, and tenth iterations of the object detection and image segmentation model allows for the rapid selection of the first target model. The tool used to annotate the target position and shape, and to convert the annotated position and shape into a data format that the model can train, is an image annotation tool, specifically a Label Img tool.

[0064] Preferably, the size of the first target model does not exceed 640 pixels, and the average precision value is not less than 37.3. The first target model is a YOLOv8n model, which has a faster inference speed than other models and can complete complex image processing and recognition tasks in a short time. In addition, this first target model also has high robustness and adaptability, and can maintain stable recognition performance under different lighting conditions, shooting angles, and object postures. Therefore, choosing the YOLOv8n model as the first target model can ensure the accuracy and real-time performance of the mouth recognition and localization method, providing a reliable foundation for subsequent operations.

[0065] Preferably, the actuator is a robotic arm. By configuring the robotic arm's actuator, the robotic arm can flexibly adjust its operating angle through its movable joints, thereby making precise movements and adjustments based on the identified mouth position. This improves the robotic arm's flexibility and accuracy, ensuring accurate arrival at the target location. During feeding, the robotic arm will fine-tune according to the real-time position and shape changes of the mouth to ensure accurate food delivery. Furthermore, the robotic arm possesses high stability and durability, maintaining stable performance during prolonged use and providing reliable service to users.

[0066] Preferably, obtaining the categorized text of the second image includes extracting image features from the second image using a visual encoder, calculating the cosine similarity between the image features and preset text features, calculating the similarity, and outputting the preset text feature corresponding to the maximum similarity. The preset text feature corresponding to the maximum similarity is the categorized text of the second image. By utilizing the feature extraction capability of the visual encoder, key information in the second image is transformed into a high-dimensional vector representation, i.e., image features. Subsequently, by calculating the cosine similarity with features in a preset text feature library, the degree of similarity between the image features and each preset text feature can be measured. In this step, by finding the preset text feature with the highest similarity to the image features and outputting it as the categorized text of the second image, the image content can be accurately matched with preset text labels, providing data support for subsequent processing and analysis.

[0067] Preferably, the construction of the embedded model includes embedding the target dataset into the initialization layer of the first target model using a multimodal pre-trained model to obtain the embedded model. The multimodal pre-trained model is a Clip model, and the embedded model is a YOLOv8n-Clip model. The weights of the target dataset acquired by the embedded model are calculated to evaluate the importance of each data item in the target dataset for the mouth recognition and localization task. This step, through weight calculation of the embedded model, automatically filters out data items highly relevant to the target task, providing a more accurate data foundation for subsequent feature extraction and model training. Simultaneously, using the Clip model as a multimodal pre-trained model fully leverages its advantages in image and text matching, effectively combining image and text information in the target dataset to improve the embedded model's ability to recognize mouth features. Furthermore, the YOLOv8n-Clip model, as the embedded model, not only inherits the performance of the YOLO series models in object detection but also further enhances the model's cross-modal understanding and recognition ability of mouth features through the introduction of the Clip model.

[0068] Preferably, the calculation of the weights of the target dataset obtained by the embedded model includes:

[0069] The weighting calculation formula is as follows:

[0070] c = c box +c i

[0071] Calculate the weights of the target dataset;

[0072] Among them, c box c represents the bounding box of the target feature region. iThis represents the entire image. `c` is the embedding vector extracted by the Clip model. Constructing an embedded model allows for rapid focusing on semantically relevant features. Calculating the weights of the target dataset improves the performance of the embedded model in handling small datasets and reduces annotation costs.

[0073] In the weight calculation formula, c box This represents the weight of the target feature region bounding box, reflecting the importance of mouth features in the image. By accurately labeling and extracting mouth feature regions, the model can focus more on this key area, thereby improving the accuracy of recognition and localization. And c i This represents the weight of the entire image, taking into account the overall information of the image, which helps the model capture contextual information related to the mouth on a global scale. By combining c... box and c i The weighting calculation formula can comprehensively evaluate the importance of each data item in the target dataset for the mouth recognition and localization task.

[0074] Furthermore, the embedding vector 'c' extracted by the Clip model serves as the basis for weight calculation, fully leveraging the Clip model's advantages in image and text matching. Through the calculation of the embedding vector, the model can effectively combine image and text information, achieving cross-modal understanding and recognition. This not only improves the model's ability to recognize mouth features but also enables the model to better adapt to mouth recognition and localization tasks under different scenarios and lighting conditions.

[0075] In summary, by calculating the weights of the target dataset obtained through the embedded model, data items highly relevant to the target task can be automatically selected, providing a more accurate data foundation for subsequent feature extraction and model training. Furthermore, combining the embedding vector calculation of the Clip model further enhances the model's cross-modal understanding and recognition capabilities of mouth features, improving the accuracy and robustness of the mouth recognition and localization method.

[0076] like Figure 2 As shown, a mouth recognition and positioning device is provided, including an embedded processor 6, a binocular camera 8, and a memory. The memory is configured to store images and executable instructions. The binocular camera 8 is configured to acquire a first image and a second image within a certain distance and send the acquired images to the memory for storage. The embedded processor 6 is configured to execute the executable instructions stored in the memory to implement the mouth recognition and positioning method as described above. This device, through the efficient computing power of the embedded processor, can process the image data acquired by the binocular camera in real time.

[0077] The binocular camera, with its left and right cameras, captures more three-dimensional and detailed mouth feature information, providing richer data support for subsequent mouth recognition and localization through binocular matching. The memory not only stores the acquired image data but also the executable instructions required to implement the mouth recognition and localization method, ensuring the automation and intelligence of the entire process. Furthermore, the device's compact design facilitates integration into various intelligent devices such as feeding robotic arms, providing strong technical support for applications requiring precise feeding.

[0078] like Figure 4 As shown, the binocular camera captures images using the left and right cameras respectively, resulting in a left camera image and a right camera image. The left camera image is the first image, and the right camera image is the second image. The left and right camera images are matched to obtain a depth map. Further data preprocessing is performed for data annotation, and the optimal model is obtained through model training and comparison. The Clip model and YOLOv8n-Clip model are used to extract and calculate the weights of the target dataset, thereby detecting the mouth prediction box, mouth depth distance, and mouth 3D coordinates for real-time mouth localization.

[0079] like Figure 5 As shown, in this embodiment, a binocular camera captures an image of a plate containing noodles, vegetables, and a poached egg. This image is then input into the Image Encoder visual encoder to obtain image features (G1). These features are then compared with all text features (T1...T1). n Calculate the cosine similarity and output the classified text: A bowl of noodles with a poached egg and greens.

[0080] like Figure 6 The diagram shown is a structural diagram of the DS18B20 temperature sensor.

[0081] like Figure 3 As shown, a feeding robotic arm is provided, including a base, a robotic arm 7, and a mouth recognition and positioning device as described above.

[0082] The embedded processor 6 is installed on one side of the base, and the power switch 1 is connected to one side of the embedded processor 6. The power switch 1 is electrically connected to the power supply. The binocular camera 8 is fixed to the top of the robotic arm 7, and a feeder 9 is axially connected to one end of the robotic arm 7.

[0083] like Figure 7As shown, a microcontroller 3 is installed at the bottom of the base. The microcontroller 3 integrates a PID temperature control system. A display screen 2 is installed on one side of the microcontroller 3, and a heating pad 5 is laid on top of the microcontroller 3. A DS18B20 temperature sensor 4 is connected to the top of the heating pad 5. The display screen 2 is a touch screen used to set the temperature threshold of the food on the plate and to display the plate temperature in real time. The microcontroller 3 is used to receive temperature data signals and convert them into digital signals to send to the display screen 2, and to output signals to the heating pad 5 through the PID temperature control system. The display screen 2 displays the temperature reading. The DS18B20 temperature sensor 4 is used to collect the temperature data signals of the plate and send them to the microcontroller 3. The embedded processor 6 controls the binocular camera 8 to recognize and locate the mouth of the person, and controls the robotic arm 7 to locate and deliver the feeder 9 to the diner's mouth according to three-dimensional coordinates. When the robotic arm 7 delivers the feeder 9 to the diner's mouth, the recognition and location of the person's mouth is completed, and feeding is achieved.

[0084] Preferably, the DS18B20 temperature sensor 4, heating pad 5, PID temperature control system, microcontroller 3, and power supply constitute an automated heating device; the DS18B20 temperature sensor 4 includes a thermistor and an analog-to-digital converter.

[0085] Preferably, the DS18B20 temperature sensor 4 has a temperature testing range of -55°C to +125°C. When the DS18B20 temperature sensor 4 is activated, it can measure the temperature of the food on the plate and directly convert the temperature into a digital signal. Then, using the 1-Wire bus protocol's single-wire communication capability, a series of pulses are transmitted to the microcontroller for parsing, converting the signal into an actual temperature value. Based on the optimal temperature range for dining (35°C to 45°C), it determines whether the food needs heating and sets a temperature control threshold as the optimal dining temperature. If the PID temperature control system detects that the food temperature is below the minimum threshold value, it activates the heating pad 5. The PID temperature control system adjusts the system output through three independent control strategies: proportional (P), integral (I), and derivative (D). The output formula of the PID temperature control system is:

[0086]

[0087] In PID, "P" represents proportional control, which adjusts the control quantity based on the difference between the target value and the actual value; "I" represents integral control, which adjusts based on the accumulation of error over time, and its output is proportional to the integral of the error; and "D" represents derivative control.

[0088] Among them, K p K represents the proportional gain, e(t) represents the current error, and the larger the error, the stronger the system's adjustment capability. iIt is the integral gain; the integral term can eliminate persistent steady-state errors, K d As a differential gain, the differential term can reduce system oscillation and overshoot. By manually controlling the three gain values, the temperature can be kept near the target temperature.

[0089] like Figure 8 As shown, by setting a plate temperature threshold, the plate temperature is measured by a temperature sensor after the meal begins. The sensor determines whether the plate temperature is greater than the temperature threshold. If it is, the plate temperature is measured again. Otherwise, the temperature difference is calculated, and the temperature is controlled by a PID control system. The temperature control signal is then sent to the automatic heating device, and the heating device is turned off after the meal is finished.

[0090] Simultaneously, the system monitors the temperature trend of the plates in real time to predict whether the heating strategy needs to be adjusted in advance. If a significant temperature drop is detected, the system will increase the heating power in advance to ensure that the food temperature is always maintained within a suitable dining range. In addition, the method includes safety monitoring of the heating device; if abnormally high temperature or short circuit is detected, the system will immediately cut off the power to ensure the user's safety.

[0091] like Figure 9 The image shown is a training result diagram of the YOLO model for mouth recognition.

[0092] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in the present invention, based on the technical solution and inventive concept of the present invention, should be covered within the scope of protection of the present invention.

Claims

1. A mouth recognition and localization method, characterized in that, include: Collect a first image and a second image within a certain distance. The first image is the opening and closing state of a human face's mouth, and the second image is a food type dataset. The target location and shape are labeled in the first image and the second image respectively, and the labeled location and shape are converted into a data format that can be trained by the model to form the first dataset and the second dataset. The first dataset is the location dataset that can be trained by the model, and the second dataset is the shape dataset that can be trained by the model. The first and second datasets are input into three iterative models respectively. Each iterative model extracts the target dataset. The first target model is obtained based on the difference between the target dataset and the expected dataset. The iterative model is an object detection and image segmentation model. The classification text of the second image is obtained and an embedded model is constructed. The classification text is collected using the embedded model to obtain the second target model. The second target model is an embedded model selected based on the acquisition speed. The second target model is an integrated model of the binocular camera. A stereo matching algorithm is used to obtain the disparity depth map of the first dataset. The disparity depth map is then input into the second target model, and the three-dimensional coordinates of the first dataset are output to the motion component. The motion component reaches the target position based on the three-dimensional coordinates.

2. The mouth recognition and positioning method according to claim 1, characterized in that: The three iterative models are the fifth, eighth, and tenth iterations of the object detection and image segmentation model, in which the object detection and image segmentation model is the YOLO model.

3. The mouth recognition and positioning method according to claim 1, characterized in that: The first target model has a size not exceeding 640 pixels and an average precision value not less than 37.

3. The first target model is a YOLOv8n model.

4. The mouth recognition and positioning method according to claim 3, characterized in that: The moving part is a robotic arm.

5. The mouth recognition and positioning method according to claim 1, characterized in that: The process of obtaining the classification text of the second image includes: A visual encoder is used to extract image features from the second image. The image features are then compared with preset text features using cosine similarity calculation. The similarity is calculated, and the preset text feature corresponding to the maximum similarity is output. The preset text feature corresponding to the maximum similarity is the classification text of the second image.

6. The mouth recognition and positioning method according to claim 3, characterized in that: The construction of the embedded model includes: The target dataset is embedded into the initialization layer of the first target model using a multimodal pre-trained model to obtain an embedded model. The multimodal pre-trained model is a Clip model, and the embedded model is a YOLOv8n-Clip model. Calculate the weights of the target dataset obtained by the embedded model.

7. The mouth recognition and positioning method according to claim 6, characterized in that: The calculation of the weights of the target dataset obtained by the embedded model includes: The weighting calculation formula is as follows: c=c box +c i Calculate the weights of the target dataset; Among them, c box c represents the bounding box of the target feature region. i This represents the entire image. `c` is the embedding vector extracted by the Clip model. Constructing an embedded model allows for rapid focusing on semantically relevant features.

8. A mouth recognition and positioning device, characterized in that, Includes embedded processors, binocular cameras, and memory; The memory is configured to store images and executable instructions; The binocular camera is configured to acquire a first image and a second image within a certain distance, and send the acquired images to the memory for storage. The embedded processor is configured to implement the mouth recognition and positioning method as described in any one of claims 1-7 when executing the executable instructions stored in the memory.

9. A feeding robotic arm, characterized in that: Includes a base, a robotic arm, and the mouth recognition and positioning device as described in the claims; The embedded processor is installed on one side of the base, and a power switch is connected to one side of the embedded processor. The power switch is electrically connected to the power source. The binocular camera is fixed to the top of the robotic arm, and a feeder is axially connected to one end of the robotic arm. A microcontroller is installed at the bottom of the base. The microcontroller integrates a PID temperature control system. A display screen is installed on one side of the microcontroller. A heating pad is laid on top of the microcontroller. A DS18B20 temperature sensor is connected to the top of the heating pad. A display screen is installed on one side of the microcontroller.

10. The feeding robotic arm according to claim 9, characterized in that: The DS18B20 temperature sensor, heating pad, PID temperature control system, microcontroller and power supply constitute an automated heating device; The DS18B20 temperature sensor includes a thermistor and an analog-to-digital converter.

Images