An image processing method, device and medium

Abstract

Embodiments of the present application provide an image processing method and device and medium, to solve the problem of inaccurate identity information recognition in the prior art when recognizing the identity of a user based on a collected image. In the embodiments of the present application, the target height, target shoulder width and target depth feature vector can be accurately determined through a first color image and a depth image. When determining the target height, target shoulder width and target depth feature vector, the determination is based on the depth information in the depth image, so even if the posture of the member being determined is different, the target height and target shoulder width can still be accurately determined, and the target identity information can be accurately determined based on the target height, target shoulder width and target depth feature vector, so the identity can be accurately recognized.

Description

Technical Field

[0001] This application relates to the field of data processing technology, and in particular to an image processing method, apparatus and medium. Background Technology

[0002] With the rise of the Internet of Things (IoT), home appliances are becoming increasingly intelligent, and smart homes are gradually covering all aspects of family life. Smart refrigerators are also seeing a surge in popularity in the high-end home appliance market. As an essential appliance, the basic functionality of smart refrigerators has largely reached its peak in this field. Improving the user experience by combining the needs of family members to make smart refrigerators more intelligent and precise has become the next breakthrough point in the high-end home appliance market. Furthermore, as a primary appliance for food management, smart refrigerators are gradually becoming an important way for users to manage the health of family members. When managing the health of family members based on their needs, smart refrigerators first identify the user's identity information and then manage the relevant information of that identified user.

[0003] In existing technologies, including smart refrigerators, identity recognition is achieved by real-time acquisition of color images, which are then used by smart devices such as refrigerators for facial recognition. However, when indoor lighting is complex, such as strong light, backlight, low light, or dim light, the image quality acquired by the image acquisition device drops sharply. This deterioration in image quality reduces the accuracy of facial recognition. Furthermore, when a face is obscured by hair or other objects, or when the angle of the face is too large, the number of effective facial feature points captured by the image acquisition device is significantly reduced, further decreasing the accuracy of facial recognition and potentially making it impossible to accurately identify the user's identity information within the image. Summary of the Invention

[0004] This application provides an image processing method, apparatus, device, and medium to solve the problem of inaccurate identity information recognition when performing user identity recognition based on acquired images in the prior art.

[0005] In a first aspect, embodiments of this application provide an image processing method, the method comprising:

[0006] Receive the first color image and depth image simultaneously acquired in the same scene;

[0007] The target shoulder key points in the first color image are obtained by using a pre-trained shoulder recognition model.

[0008] Based on the target shoulder key points and the depth image, determine the target height and target shoulder width; based on the number of pixels in each preset depth interval in the depth image, determine the target depth feature vector;

[0009] Based on the correspondence between height, shoulder width, and depth feature vectors and identity information, the target identity information corresponding to the target height, target shoulder width, and target depth feature vectors is determined.

[0010] Secondly, embodiments of this application also provide an image processing apparatus, the apparatus comprising:

[0011] The receiving and acquisition module is used to receive a first color image and a depth image simultaneously acquired in the same scene; and to acquire the target shoulder key points in the first color image through a pre-trained shoulder recognition model.

[0012] The processing module is used to determine the target height and target shoulder width based on the target shoulder key points and the depth image; determine the target depth feature vector based on the number of pixels in each preset depth interval in the depth image; and determine the target identity information corresponding to the target height, the target shoulder width, and the target depth feature vector based on the correspondence between the height, shoulder width, and depth feature vector and identity information.

[0013] Thirdly, embodiments of this application also provide an electronic device, which includes at least a processor and a memory, wherein the processor is used to execute a computer program stored in the memory to implement the steps of any of the image processing methods described above.

[0014] Fourthly, embodiments of this application also provide a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of any of the image processing methods described above.

[0015] In this embodiment of the application, the target height, target shoulder width, and target depth feature vector can be accurately determined using the first color image and the depth image. Since the target height, target shoulder width, and target depth feature vector are determined based on the depth information in the depth image, the target height and target shoulder width can still be accurately determined even if the posture of the member being determined is different. Thus, the target identity information can be accurately determined based on the target height, target shoulder width, and target depth feature vector, thereby enabling accurate identity recognition. Attached Figure Description

[0016] To more clearly illustrate the technical solutions of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0017] Figure 1 This is a schematic diagram of an image processing process provided in an embodiment of this application;

[0018] Figure 2 A schematic diagram of the viewpoint of the image acquisition device for acquiring depth images provided in the embodiments of this application;

[0019] Figure 3 A schematic diagram of a depth image after determining key points of the target shoulder, provided for an embodiment of this application;

[0020] Figure 4 This is a schematic diagram illustrating the identity recognition process provided in an embodiment of this application;

[0021] Figure 5 A schematic diagram illustrating the installation of an image acquisition device for acquiring color and depth images on a smart refrigerator, as provided in an embodiment of this application.

[0022] Figure 6 Detailed schematic diagrams illustrating intelligent management of different identity information provided for embodiments of this application;

[0023] Figure 7 A schematic diagram illustrating the process of determining relevant information of the target ingredient as provided in the embodiments of this application;

[0024] Figure 8 A detailed flowchart illustrating the identity recognition process provided in this application embodiment;

[0025] Figure 9 This is a schematic diagram of an image processing device structure provided in an embodiment of this application;

[0026] Figure 10 This is a schematic diagram of an electronic device structure provided in an embodiment of this application. Detailed Implementation

[0027] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art are within the scope of protection of this application.

[0028] In this embodiment of the application, in order to accurately identify the target, a first color image and a depth image simultaneously acquired in the same scene are received. The first color image is input into a pre-trained shoulder recognition model to obtain the target shoulder key points in the first color image. Based on the obtained target shoulder key points and the depth image simultaneously acquired in the same scene as the first color image, the target height and target shoulder width are determined. Based on the number of pixels in each preset depth interval in the depth image, the target depth feature vector can be determined. After determining the target height, target shoulder width, and target depth feature vector, the target identity information corresponding to the target height, target shoulder width, and target depth feature vector is determined based on the correspondence between the height, shoulder width, and depth feature vector and the identity information, thereby accurately realizing the identification of the target.

[0029] Figure 1 This application provides a schematic diagram of an image processing process, which includes the following steps:

[0030] S101: Receive the first color image and depth image acquired simultaneously in the same scene.

[0031] The image processing method provided in this application is applied to an electronic device, which may be a smart refrigerator, PC, or server, or other smart devices.

[0032] In this embodiment, identity recognition is performed on the received first color image and depth image. To ensure accurate identity recognition, the first color image and depth image are images of the same scene captured at the same time. Furthermore, in this embodiment, the first color image and depth image can be captured using an RGBD camera, and the captured images are of the same size; alternatively, the first color image can be captured using an RGB camera, and the depth image can be captured using a depth camera, and the captured images can be adjusted to the same size. In this embodiment, the first color image and depth image can be captured by the image acquisition device after the electronic device determines that the smart refrigerator has been opened. Specifically, how to determine whether the smart refrigerator has been opened is existing technology and will not be elaborated here.

[0033] S102: Obtain the target shoulder key points in the first color image using a pre-trained shoulder recognition model.

[0034] In this embodiment, after receiving the first color image and the depth image, in order to accurately perform identity recognition, the received first color image can be input into a pre-trained shoulder recognition model to obtain the output of the shoulder recognition model. The output of the shoulder recognition model is the target shoulder key points in the first color image, specifically the position information of the target shoulder key points in the first color image. Furthermore, the target shoulder key points are two shoulder key points, namely the key points of the left and right shoulders of the human body (where left and right refer to the left and right sides of the human body in reality).

[0035] Furthermore, in this embodiment, to accurately obtain the target shoulder key points, the region containing the human body in the image can be pre-selected as the region of interest, or the region containing the shoulder in the image can be pre-selected as the region of interest. This allows for region of interest extraction of the acquired first color image, cropping the target region from the first color image. After cropping the target region, the cropped target region is input into a pre-trained shoulder recognition model to obtain the target shoulder key points contained within the target region, thereby significantly shortening the processing time. Specifically, how to perform region of interest extraction is existing technology and will not be elaborated upon here.

[0036] S103: Determine the target height and target shoulder width based on the target shoulder key points and the depth image; determine the target depth feature vector based on the number of pixels in each preset depth interval in the depth image.

[0037] Since the depth image and the first color image are images of the same scene captured at the same time, and since they are images of the same size, there is a one-to-one correspondence between the pixels in the depth image and the first color image. Therefore, after determining the target shoulder keypoint in the first color image, the position of the target shoulder keypoint in the depth image can be determined. Based on the position of the target shoulder keypoint in the depth image, the target height and target shoulder width can be determined. Because the image acquisition device for acquiring the depth image is usually installed on top of the smart refrigerator, it is generally difficult for an adult's head to be covered by the forward observer vehicle (FOV) of the image acquisition device, meaning it is difficult to directly determine the vertical distance from the top of the head to the ground. Therefore, in this embodiment, the height of the member's shoulder is determined by directly calculating the vertical distance from the shoulder to the ground. The height of the shoulder represents the member's height, i.e., the target height in this embodiment. Furthermore, the determined shoulder height can be the vertical distance from the center point of the member's shoulder to the ground, or it can be the vertical distance from any shoulder keypoint of the member to the ground.

[0038] Furthermore, in this embodiment, multiple preset depth intervals are pre-stored. Even if the same member stands in different postures within the application scenario, the distribution of histogram bars and peak variations in the depth histogram corresponding to the acquired depth images remain consistent. Moreover, the horizontal axis of the depth histogram represents the depth value, and the vertical axis represents the number of pixels. This means that the number of pixels for the same member is consistent across different depth values; that is, the number of pixels within the preset depth intervals in the acquired depth images of the same member is consistent. Therefore, in this embodiment, the target depth feature vector can be determined based on the number of pixels in each preset depth interval of the depth image.

[0039] S104: Based on the correspondence between height, shoulder width, and depth feature vectors and identity information, determine the target identity information corresponding to the target height, the target shoulder width, and the target depth feature vector.

[0040] Since different members have different height, shoulder width, and depth feature vectors, and the electronic device pre-stores the correspondence between height, shoulder width, and depth feature vectors and identity information, the identity information of family members can be determined based on differences in human physiological structure in this embodiment. Therefore, after determining the target height, target shoulder width, and target depth feature vectors, the height, shoulder width, and depth feature vectors that match the target height, target shoulder width, and target depth feature vectors are determined, and the identity information corresponding to the matched height, shoulder width, and depth feature vectors is determined as the target identity information. Specifically, the correspondence between height, shoulder width, and depth feature vectors and identity information can be stored in a human three-dimensional database in the electronic device.

[0041] In this embodiment, since the determined target height, target shoulder width, and target depth feature vectors may have some deviation, a classifier can be used to match whether the features are identical when determining which of the height, shoulder width, and depth feature vectors stored in the electronic device is the target height, target shoulder width, and target depth feature vector. This classifier can be an SVM. The trained classifier determines the height, shoulder width, and depth feature vectors that match the target height, target shoulder width, and target depth feature vectors. Based on the correspondence between the stored height, shoulder width, and depth feature vectors and identity information, the trained classifier directly outputs the predicted identity information and confidence level. If the predicted confidence level is greater than a preset threshold, the output result is considered credible, and the identity information is determined to be the target identity information. Specifically, how the classifier predicts identity information and confidence level is existing technology and will not be elaborated here.

[0042] In this embodiment of the application, the target height, target shoulder width, and target depth feature vector can be accurately determined using the first color image and the depth image. Since the target height, target shoulder width, and target depth feature vector are determined based on the depth information in the depth image, the target height and target shoulder width can still be accurately determined even if the posture of the member being determined is different. Thus, the target identity information can be accurately determined based on the target height, target shoulder width, and target depth feature vector, thereby enabling accurate identity recognition.

[0043] To improve the accuracy of identity recognition, based on the above embodiments, in this embodiment, determining the target height and target shoulder width based on the target shoulder key points and the depth image includes:

[0044] Determine the target pixel points corresponding to the key points of the target shoulder in the depth image;

[0045] The target height and target shoulder width are determined based on the actual coordinates of each pixel in the depth image and the target pixel.

[0046] In this embodiment, when determining the target height and shoulder width, the target pixels corresponding to the key points of the target shoulder in the depth image are determined. Since the first color image and the depth image are the same size, each pixel in the first color image and the depth image corresponds one-to-one. Furthermore, since the first color image and the depth image are images of the same scene acquired at the same time, the content acquired in the first color image and the depth image is the same. The position of the same object in the first color image is consistent with its position in the depth image. Therefore, after determining the key points of the target shoulder in the first color image, the corresponding target pixels in the depth image can be determined. And since there are two key points on the target shoulder, the determined target pixels are two target pixels.

[0047] After determining the target pixel in the depth image, since each pixel in the depth image corresponds to an actual coordinate, the actual coordinates of the target pixel in the depth image can be determined. Based on the target pixel, the target height and target shoulder width can be determined.

[0048] Specifically, the actual coordinates corresponding to the target pixel can be defined with the image acquisition device acquiring the depth image as the origin, the positive x-axis as the direction forward (where forward and backward refers to front and back in reality) along the image acquisition device, the positive y-axis as the direction perpendicular to the ground and downward, and the positive z-axis as the direction perpendicular to both the x and y axes and to the right (where left and right refers to left and right in reality) along the image acquisition device. When determining the target height, based on the two determined target pixels, the average y-value of the two coordinate distances corresponding to the two target pixels can be determined, and the pre-saved distance value between the image acquisition device and the ground can be obtained. The difference between this distance value and the determined average y-value is the height of the member's shoulders. In this embodiment, the member's shoulder height is used to represent the member's height, that is, the determined difference is the target height. Of course, the difference between the pre-saved distance value and any y-value of the corresponding two coordinate distances can also be determined as the target height. Specifically, how to determine the target height is not limited here. When determining the target shoulder width, the difference in z-values ​​between the two coordinate distances corresponding to two target pixels can be determined. After determining the difference, the absolute value of the difference is determined as the target shoulder width.

[0049] For example, if the actual coordinates of two target pixels identified in a depth image are Pointleft(x1,y1,z1) and PointRight(x2,y2,z2), and the distance between the image acquisition device and the ground is y3, then the target height can be y3-(y1+y2) / 2, and the target shoulder width can be |z1-z2|.

[0050] In this embodiment of the application, the image acquisition device for acquiring depth images is typically installed on top of the smart refrigerator. Figure 2 A schematic diagram of the viewpoint of an image acquisition device for acquiring depth images provided in an embodiment of this application.

[0051] Depend on Figure 2 As is known, image acquisition devices for capturing depth images are typically installed on top of smart refrigerators. The head of an average adult is generally difficult to cover within the field of view (FOV) of such devices, making it difficult to directly determine the vertical distance from the top of the head to the ground. Therefore, in this embodiment, the vertical distance from the shoulders to the ground is directly calculated. Then, the vertical distance from the center points of the left and right shoulder keypoints to the ground is calculated. This vertical distance from the center points of the left and right shoulder keypoints to the ground is the shoulder height of the member in the acquired image, which is the target height determined in this embodiment.

[0052] Figure 3 This is a schematic diagram of a depth image after determining key points on the target shoulder, as provided in an embodiment of this application.

[0053] Depend on Figure 3As can be seen, the target pixels corresponding to the key points of the target's shoulder can be easily determined in the depth image, thereby accurately and effectively determining the target's height and shoulder width.

[0054] To accurately identify individuals, based on the above embodiments, the method in this application embodiment further includes:

[0055] Face recognition is performed on the first color image. If candidate identity information is identified through the first color image and the candidate identity information is inconsistent with the target identity information, the target identity information is updated using the candidate identity information.

[0056] In this embodiment, to prevent inaccurate identity recognition, facial recognition is used to further ensure its accuracy. Specifically, if candidate identity information corresponding to a face is identified, it is determined whether the candidate identity information identified by facial recognition is consistent with the target identity information determined above. If the candidate identity information identified by facial recognition is consistent with the target identity information, it indicates that the determined target identity information is accurate, and no changes are needed. If the candidate identity information identified by facial recognition is inconsistent with the target identity information, the determined target identity information is updated using the candidate identity information identified by facial recognition. Furthermore, if no candidate identity information is identified by facial recognition, no updates to the target identity information are needed.

[0057] The specific process of recognizing the first color image by face recognition is as follows: the first color image is input into a deep learning network, which can be FaceBoxes. The trained deep learning network is used to perform face detection to obtain multiple face box positions and corresponding confidence scores (Conf_facei). After obtaining multiple face box positions and corresponding confidence scores, the electronic device selects face boxes with confidence scores Conf_facei greater than a preset confidence threshold for retention and outputs the face box position information.

[0058] Then, based on the face bounding box position information, the first color image is cropped to obtain an image containing only the face. This cropped image is then input into a facial landmark detection network to obtain the position information of facial landmarks such as the left eyeball, right eyeball, nose tip, left corner of mouth, and right corner of mouth from the cropped image output by the network. The electronic device performs face alignment based on the detected facial landmark position information, that is, transforming the key position information points of the face image to be recognized to the position information of known key points in a standard image, obtaining a transformation matrix. The specific process of obtaining this transformation matrix is ​​existing technology and will not be elaborated here. Based on the obtained transformation matrix, the face image to be recognized is then comprehensively corrected.

[0059] Finally, the corrected face image is input into a face recognition deep learning network. The trained network extracts features of the face to be recognized, obtaining a set of face features. Principal Component Analysis (PCA) is then used to reduce the dimensionality of these features, ultimately yielding a set of target face features. A similarity metric is used to measure the similarity between the obtained target face features and features stored in the face registration database of the electronic device. This similarity metric can be cosine similarity. The extracted target face features are compared one-to-one with the features in the face registration database to determine the similarity. Features with the highest similarity, exceeding a preset threshold, are selected. Based on the pre-saved correspondence between features and identity information, the identity information corresponding to these features is determined as the candidate identity information for face recognition. Specifically, the face recognition method used is existing technology and will not be elaborated upon here.

[0060] Figure 4 This is a schematic diagram illustrating the identity recognition process provided in an embodiment of this application.

[0061] Depend on Figure 4 As can be seen, depth images and a first color image are acquired using an RGBD camera. Face detection and recognition are performed using the first color image. After facial features are identified, they are compared with features stored in the electronic device to determine the face recognition result as candidate identity information. The target's height, shoulder width, and depth feature vectors are determined using the first color image and the depth image. Based on the correspondence between height, shoulder width, and depth feature vectors and identity information stored in the electronic device, the target's identity information corresponding to these feature vectors is determined. When candidate identity information is identified through face recognition, and the candidate identity information differs from the target identity information, the face recognition result prevails. That is, if the candidate identity information differs from the target identity information, the candidate identity information is used to update the target identity information. When candidate identity information is not identified through face recognition, but the target identity information is determined through the first color image and the depth image, the target identity information determined through the first color image and the depth image prevails. That is, the target identity information is not updated. In other words, if candidate identity information is determined through facial recognition and target identity information is determined through the first color image and depth image, then the candidate identity information determined by facial recognition shall prevail. If candidate identity information is not determined through facial recognition, or target identity information is not determined through the first color image and depth image, then the determined identity information shall prevail.

[0062] Furthermore, in order to accurately identify candidate identity information through face recognition, in this embodiment of the application, face recognition can be performed together with other color images acquired simultaneously with the first color image.

[0063] Figure 5 This is a schematic diagram of the installation of an image acquisition device for acquiring color and depth images on a smart refrigerator, as provided in an embodiment of this application.

[0064] In this embodiment of the application, the image acquisition device for acquiring color images, namely an RGB camera, can be set as follows: Figure 5 The locations shown indicate that the RGB camera can be placed on the sides and top of the smart refrigerator. By capturing color images from multiple angles, facial recognition can be performed more accurately. Furthermore, the installation location of the RGB camera is not limited to... Figure 5 The three positions in the middle, Figure 5 The RGB cameras on both sides can be used to recognize incomplete faces or to acquire facial key point information when the face angle changes. While the recognition accuracy of the side RGB cameras is not as high as that of the top RGB camera, their unique positional advantage still helps in face recognition at different angles. The depth camera can be installed at the very top of the smart refrigerator, such as... Figure 5 The location shown.

[0065] To improve user experience, based on the above embodiments, the method in this application embodiment further includes:

[0066] If the identity information corresponding to the target height, target shoulder width, and target depth feature vector is not saved, and the face recognition of the first color image fails to identify the candidate identity information, then a preset first reminder message will be output.

[0067] Since the identity information corresponding to the target height, shoulder width, and depth feature vectors may not be recorded, and face recognition cannot determine the candidate identity information, meaning that the electronic device does not have height, shoulder width, and depth feature vectors that match the target height, shoulder width, and depth feature vectors, and face recognition of the first color image fails to determine the candidate identity information, a preset first reminder message is output to improve the user experience. This first reminder message can be "The current member has not registered." Furthermore, in this embodiment, if the electronic device does not store the identity information corresponding to the target height, shoulder width, and depth feature vectors and has not recorded, and face recognition cannot determine the candidate identity information, it indicates that the current user has not registered. Therefore, the first reminder message can also be "You have not registered. Do you want to register?"

[0068] If a user's registration request is received, the identity information contained in the registration request is obtained, and a correspondence is established between the target height, target shoulder width, and target depth feature vectors determined above and the identity information, as well as a correspondence between the determined facial features and the identity information. That is, if it is determined that the pre-saved correspondence between height, shoulder width, and depth feature vectors and identity information does not contain a height, shoulder width, and depth feature vector matching the target height, target shoulder width, and target depth feature vectors, and no matching feature is found in the facial registration database, a first reminder message can be output to remind the member to register. If a member's registration request is received, the correspondence between the target height, target shoulder width, and target depth feature vectors and the identity information carried in the registration request is determined, and this correspondence is saved in the electronic device. Facial features are extracted using the facial recognition method described above, a correspondence is established between the extracted facial features and the identity information carried in the registration request, and this correspondence is saved in the electronic device.

[0069] To enhance user experience, based on the above embodiments, the method in this application embodiment further includes:

[0070] Obtain a second color image; using a pre-trained food recognition model, obtain the name and quantity of the target food in the second color image;

[0071] Based on the pre-saved correspondence between food names and nutrients, determine the nutritional components corresponding to the target food name; based on the quantity of the target food and the nutritional components, determine the target nutritional components.

[0072] In this embodiment, after determining the identity information of the user accessing the food ingredients, the relevant information of the accessed food ingredients is managed. Specifically, when determining the relevant information of the accessed food ingredients, a second color image can be acquired first. The acquired second color image is then input into a pre-trained food ingredient recognition model, and the output of the food ingredient recognition model is determined to be the target food ingredient name and the target food ingredient quantity. How to determine the target food ingredient name and the target food ingredient quantity in the image using the food ingredient recognition model is existing technology and will not be elaborated here.

[0073] Since some members may have specific requirements for certain nutrients, in this embodiment of the application, the electronic device stores the correspondence between food ingredients and nutrients in advance. After determining the name of the target food ingredient, the nutrient corresponding to the name of the target food ingredient is determined. Since there is more than one target food ingredient, the product of the determined nutrient and the determined number of target food ingredients is the target nutrient.

[0074] To enhance user experience, based on the above embodiments, the method in this application embodiment further includes:

[0075] If the target identity information is a first type of preset identity information, then the forbidden food corresponding to the saved target identity information is obtained; if the target food is any forbidden food corresponding to the target identity information, then the preset second reminder information is output.

[0076] If the target identity information is the second type of preset identity information, then obtain the contraindicated ingredients and contraindicated content values ​​corresponding to the saved target identity information, obtain the target content value of the contraindicated ingredients contained in the target nutrient, and if the target content value exceeds the contraindicated content value, then output the preset third reminder information.

[0077] In this embodiment of the application, in order to achieve intelligent management, prohibited ingredients or prohibited components are pre-stored for different identity information. If prohibited components are stored, the prohibited content value of the prohibited components can also be stored. Members of certain identity information may have prohibited foods, while members of other identity information may have nutrients that need to be consumed in small amounts. Therefore, the electronic device pre-stores a first type of preset identity information and a second type of preset identity information. The first type of preset identity information is for identities with prohibited ingredients, and the second type of preset identity information is for identities with prohibited components. Furthermore, the prohibited content value of the prohibited components is also stored for the second type of preset identity information. This prohibited content value can refer to the remaining edible content value within a preset time period.

[0078] If the target identity information is the first type of preset identity information, then the prohibited ingredients corresponding to that target identity information are obtained. These prohibited ingredients may include multiple ingredients; therefore, if the target ingredient is any of the prohibited ingredients, it means that the selected target ingredient is not suitable for the member with that target identity information. Therefore, a preset second reminder message is output to remind the member with that target identity information. This second reminder message could be: "The current ingredient is harmful to your health; please consider carefully."

[0079] If the target identity information is the second type of preset identity information, then the prohibited ingredients and their content values ​​corresponding to that target identity information are obtained, and the target content value of the prohibited ingredients contained in the target nutritional components is also obtained. It is determined whether the target content value of the prohibited ingredients in the target food exceeds the prohibited content value. If it exceeds the prohibited content value, it means that the selected target food is not suitable for consumption by the member with that target identity information. Therefore, a preset third reminder message is input to remind the member with that target identity information. The third reminder message can be "The current food is not suitable for your consumption".

[0080] Figure 6 This is a detailed schematic diagram illustrating intelligent management of different identity information provided in the embodiments of this application.

[0081] Depend on Figure 6 As can be seen, after the electronic device determines that the smart refrigerator has been opened, it controls the image acquisition device to acquire images and performs identity recognition and food identification based on the acquired images. Specifically, after identifying the target identity information, if the identified identity information is a first-class preset identity information, such as a child in the family, it determines whether the pre-saved list of prohibited foods for children includes the target food. In other words, it determines whether the prohibited foods corresponding to the target identity information include the target food. For example, when a child takes out sweets, fast-moving consumer goods, etc., a prompt is given to the child via voice broadcast. Furthermore, in this embodiment, information can also be sent to a preset member, which can be a parent, thereby achieving a real-time monitoring purpose.

[0082] Furthermore, after identifying the target's identity information, if the identified identity information is a first-class preset identity information, such as a family member with a chronic illness, like a grandfather, then it determines whether the retrieved target food is included in the list of prohibited foods corresponding to that target identity information. If the retrieved target food is any of the prohibited foods corresponding to that target identity information, a reminder is given via voice broadcast. It can also send messages to family members; specifically, reminders can be sent to family members through the app.

[0083] Furthermore, after identifying the target's identity information, if the target's identity information is identified as the first type of preset identity information, such as a pregnant woman in the family, then it is determined whether the prohibited food items corresponding to the target's identity information include the target food item to be taken out. If the target food item to be taken out is any of the prohibited food items corresponding to the target's identity information, a reminder is given through voice broadcast, and a message can also be sent to family members. Specifically, family members can be reminded through the APP.

[0084] Furthermore, after identifying the target identity information, if the target identity information is identified as the second type of preset identity information, such as a family member with diabetes, the system obtains the contraindicated ingredients and their content values ​​corresponding to the target identity information, and obtains the target content value of the contraindicated ingredients contained in the target nutrient. It then determines whether the target content value of the contraindicated ingredients in the target food exceeds the contraindicated content value. If it does, a voice broadcast will be used to remind the user, and a message can also be sent to family members. Specifically, family members can be reminded through the APP.

[0085] To enhance user experience, based on the above embodiments, in this embodiment, if the target ingredient is determined to be any prohibited ingredient corresponding to the target identity information, before outputting the preset second reminder information, the method further includes:

[0086] If a fasting time range is stored for the target identity information, then the stored fasting time range corresponding to the target identity information is obtained, and it is determined whether the time when the target food is taken out is within the fasting time range. If so, the operation of outputting the preset second reminder information is executed.

[0087] In this embodiment, to achieve intelligent management, prohibited foods are pre-stored for different identity information, and when prohibited foods are stored, a corresponding fasting time range may also be stored. That is, the first type of pre-set identity information may also correspond to a fasting time range. This fasting time range could be for members who want to lose weight, and these members should avoid high-calorie, high-carbohydrate foods at night. However, if the target food is something like fruit, it will not hinder weight loss.

[0088] Therefore, in this embodiment, if the target identity information is a first type of preset identity information, and if it is determined that the retrieved target food is any prohibited food corresponding to the target identity information, it is determined whether the target identity information corresponds to a stored fasting time range. If a fasting time range is stored, it means that the member of the target identity information is not suitable to eat the target food within the fasting time range. Therefore, if the target identity information corresponds to a stored fasting time range, the fasting time range corresponding to the target identity information is obtained. If the time when the target food is retrieved is within the fasting time range corresponding to the target identity information, it means that the member of the target identity information retrieved an unsuitable food within the fasting time range, and a preset second reminder message is output to remind the member of the target identity information. The second reminder message can be "Currently it is your fasting time".

[0089] Furthermore, in this embodiment of the application, after determining the target ingredient, the recipe database can be pre-stored to determine the dishes that can be made with the target ingredient and the corresponding recipes. The recipes that can be made with the target ingredient are then sent to the APP corresponding to the target identity information.

[0090] In order to manage food information, based on the above embodiments, the method in this application embodiment further includes:

[0091] Receive each frame of color image, as well as the depth image acquired simultaneously in the same scene as each frame of color image; determine the target depth image to be acquired when storing or retrieving food based on each frame of color image and the corresponding depth image.

[0092] Based on the actual distance corresponding to each pixel in the target depth image and the pre-saved correspondence between the storage location and the actual distance range, the location where the ingredients are stored is determined.

[0093] The stored ingredient information is managed based on the target ingredient name, the target ingredient quantity, the target nutritional components, and the storage location.

[0094] In this embodiment of the application, in order to accurately manage the food information, each frame of color image and the depth image simultaneously acquired in the same scene as each frame of color image can be received first. Based on each frame of color image and the corresponding depth image, the target depth image acquired when storing and retrieving food can be determined.

[0095] The specific process for determining the target depth image is as follows: Since the positions of the ingredients are stored and retrieved are different, the final position of the hand will also be different. Therefore, in this embodiment, for each received color image frame, the position range of the hand in that color image frame is determined. Based on the determined position range of the hand in that color image frame and the depth image corresponding to that color image frame, the position range of the hand in the depth image corresponding to that color image frame is determined. The actual coordinates of the pixels within that position range are determined, thereby determining the actual coordinates of the pixels within the position range of the hand in each depth image frame.

[0096] Since the hand is closest to the image acquisition device capturing the depth image in a specific preset direction when storing or retrieving food, and the depth image captured by the image acquisition device when the hand retrieves food is the target depth image, the average distance in that preset direction corresponding to the actual coordinates of the pixels within the hand's location range in each frame of depth image can be determined. The frame with the smallest determined average distance is the depth image captured by the image acquisition device when the hand stores or retrieves food, which is the target depth image mentioned in this application embodiment. Therefore, the target depth image corresponding to storing or retrieving food can be determined.

[0097] Specifically, if the image acquisition device that acquires depth images is taken as the origin of the coordinate system, and the direction forward (where forward and backward refers to front and back in reality) along the image acquisition device is taken as the positive x-axis direction, and if this preset direction is the x-axis direction, then when determining the target depth image, the average distance value in the x-axis direction of the actual coordinates of the pixels within the position range of the hand in each frame of the depth image can be determined, and the frame of the depth image with the smallest average value is the target depth image.

[0098] Because the location of the food items varies, the distance between the hand and the image acquisition device that captures the depth image also varies when storing or retrieving the food. Therefore, the electronic device pre-stores the correspondence between the storage location and the actual distance range. In this embodiment, the location where the food items are stored can be determined based on the actual distance corresponding to each pixel in the determined target depth image and the pre-stored correspondence between the storage location range and the actual distance. Specifically, if the actual coordinates of the pixel are taken with the image acquisition device capturing the depth image as the origin and the vertical downward direction between the image acquisition device and the ground as the positive y-axis, then the actual distance can refer to the distance corresponding to this y-axis direction.

[0099] In this embodiment, after determining the target ingredient's name, quantity, nutritional components, storage location, and specific access actions, the stored ingredient information is managed. The method for determining the corresponding access actions for each ingredient is prior art and will not be elaborated here. Furthermore, in this embodiment, the acquired ingredient-related information includes, but is not limited to, ingredient name, ingredient type, nutritional components, quantity, ingredient image, and suitable population. Specifically, the suitable population for an ingredient can be determined by pre-saving a correspondence between ingredient names and suitable populations; after identifying the target ingredient, the suitable population corresponding to the target ingredient's name is then determined.

[0100] In this embodiment, the user database includes the following identity information: basic information, identification information, health information, and dietary history information. Basic information includes name, gender, age, etc.; identification information includes information about family members, such as grandparents, whether they belong to the first type of preset identity information, whether they belong to the second type of preset identity information, etc.; health information covers the family user's disease information and dietary restrictions; dietary history records the time each family user used the refrigerator and the history of stored ingredients. The food database includes the following information: ingredient name, ingredient type, nutritional components, ingredient quantity, ingredient image, access information, etc. Here, access information records access time, access location, access judgment, etc.

[0101] Furthermore, in this embodiment of the application, the target shoulder key points can be determined by the acquired depth image and the shoulder recognition model. In order to accurately obtain the target shoulder key points, the region where the human body is located in the image can be selected as the region of interest in advance, or the region where the shoulder is located in the image can be selected as the region of interest, thereby extracting the region of interest from the depth image, cropping the target region in the depth image, and after cropping the target region, inputting the cropped target region into the pre-trained shoulder recognition model to obtain the target shoulder key points contained in the target region.

[0102] Figure 7 This is a schematic diagram illustrating the process of determining relevant information of the target ingredient as provided in the embodiments of this application.

[0103] Depend on Figure 7 It can be seen that after the electronic device determines that the smart refrigerator has been turned on, the electronic device controls the image acquisition device to acquire images. The electronic device uses the images acquired by the image acquisition device to detect and identify food ingredients, obtains relevant information about the target food ingredients, and adds the obtained relevant information about the target food ingredients to the food ingredient database.

[0104] Figure 8 This is a detailed flowchart illustrating the identity recognition process provided in an embodiment of this application.

[0105] Depend on Figure 8It is understood that when the smart refrigerator is detected to be opened, the electronic device controls the RGBD camera to start acquiring images. The user's identity features are extracted from the first color image and depth image. These extracted features are compared with the features corresponding to registered identity information in the database to determine the target's identity. These features refer to height, shoulder width, and depth feature vectors, and can also refer to facial features. If the target's identity is not determined, the user is asked to register. If registration is required, the system receives the user's actively inputted new identity information and saves the correspondence between the identified features and this new identity information to the database. In other words, it saves the correspondence between the identified height, shoulder width, and depth feature vectors and the new identity information, as well as the correspondence between facial features and the new identity information. If the target's identity has been detected, the target's identity information and its corresponding food access record are saved to the user database. This food access record includes information such as food name, quantity, nutritional content, access action, and access location.

[0106] Figure 9 This application provides a schematic diagram of an image processing apparatus, which includes:

[0107] The receiving and acquisition module 901 is used to receive a first color image and a depth image simultaneously acquired in the same scene; and to acquire the target shoulder key points in the first color image through a pre-trained shoulder recognition model.

[0108] The processing module 902 is used to determine the target height and target shoulder width based on the target shoulder key points and the depth image; determine the target depth feature vector based on the number of pixels in each preset depth interval in the depth image; and determine the target identity information corresponding to the target height, the target shoulder width and the target depth feature vector based on the correspondence between the height, shoulder width and depth feature vector and identity information.

[0109] In one possible implementation, the processing module 902 is specifically used to determine the target pixel point corresponding to the target shoulder key point in the depth image; and to determine the target height and target shoulder width based on the actual coordinates of each pixel point in the depth image and the target pixel point.

[0110] In one possible implementation, the processing module 902 is further configured to perform face recognition on the first color image, and if candidate identity information is identified through the first color image, and the candidate identity information is inconsistent with the target identity information, then the target identity information is updated using the candidate identity information.

[0111] In one possible implementation, the processing module 902 is further configured to output a preset first reminder message if the identity information corresponding to the target height, the target shoulder width, and the target depth feature vector is not saved, and the candidate identity information is not determined by face recognition of the first color image.

[0112] In one possible implementation, the processing module 902 is further configured to acquire a second color image; acquire the target ingredient name and the quantity of the target ingredient in the second color image using a pre-trained ingredient recognition model; determine the nutritional components corresponding to the target ingredient name based on a pre-saved correspondence between ingredient names and nutrients; and determine the target nutritional components based on the quantity of the target ingredient and the nutritional components.

[0113] In one possible implementation, the processing module 902 is further configured to: if the target identity information is a first type of preset identity information, then obtain the prohibited food corresponding to the saved target identity information; if the target food is any prohibited food corresponding to the target identity information, then output a preset second reminder message; if the target identity information is a second type of preset identity information, then obtain the prohibited components and prohibited content values ​​corresponding to the saved target identity information, obtain the target content value of the prohibited components contained in the target nutrient, and if the target content value exceeds the prohibited content value, then output a preset third reminder message.

[0114] In one possible implementation, the processing module 902 is further configured to, if a fasting time range is stored corresponding to the target identity information, obtain the fasting time range stored corresponding to the target identity information, determine whether the time when the target food is taken out is within the fasting time range, and if so, perform the subsequent operation of outputting a preset second reminder message.

[0115] In one possible implementation, the processing module 902 is further configured to receive each frame of color image and a depth image simultaneously acquired in the same scene as each frame of color image; determine the target depth image acquired when storing or retrieving ingredients based on each frame of color image and the corresponding depth image; determine the location where the ingredients are stored based on the actual distance corresponding to each pixel in the target depth image and the pre-saved correspondence between the storage location and the actual distance range; and manage the stored ingredient information based on the target ingredient name, the target ingredient quantity, the target nutritional components, and the storage location.

[0116] Figure 10 This application provides a schematic diagram of an electronic device structure. Based on the above embodiments, this invention also provides an electronic device, such as... Figure 10As shown, it includes: processor 1001, communication interface 1002, memory 1003 and communication bus 1004, wherein processor 1001, communication interface 1002 and memory 1003 communicate with each other through communication bus 1004.

[0117] The memory 1003 stores a computer program, which, when executed by the processor 1001, causes the processor 1001 to perform the following steps:

[0118] Receive the first color image and depth image simultaneously acquired in the same scene;

[0119] The target shoulder key points in the first color image are obtained by using a pre-trained shoulder recognition model.

[0120] Based on the target shoulder key points and the depth image, determine the target height and target shoulder width; based on the number of pixels in each preset depth interval in the depth image, determine the target depth feature vector;

[0121] Based on the correspondence between height, shoulder width, and depth feature vectors and identity information, the target identity information corresponding to the target height, target shoulder width, and target depth feature vectors is determined.

[0122] In one possible implementation, determining the target height and target shoulder width based on the target shoulder key points and the depth image includes:

[0123] Determine the target pixel points corresponding to the key points of the target shoulder in the depth image;

[0124] The target height and target shoulder width are determined based on the actual coordinates of each pixel in the depth image and the target pixel.

[0125] In one possible implementation, the method further includes:

[0126] Face recognition is performed on the first color image. If candidate identity information is identified through the first color image and the candidate identity information is inconsistent with the target identity information, the target identity information is updated using the candidate identity information.

[0127] In one possible implementation, the method further includes:

[0128] If the identity information corresponding to the target height, target shoulder width, and target depth feature vector is not saved, and the face recognition of the first color image fails to identify the candidate identity information, then a preset first reminder message will be output.

[0129] In one possible implementation, the method further includes:

[0130] Obtain a second color image; using a pre-trained food recognition model, obtain the name and quantity of the target food in the second color image;

[0131] Based on the pre-saved correspondence between food names and nutrients, determine the nutritional components corresponding to the target food name; based on the quantity of the target food and the nutritional components, determine the target nutritional components.

[0132] In one possible implementation, the method further includes:

[0133] If the target identity information is a first type of preset identity information, then the forbidden food corresponding to the saved target identity information is obtained; if the target food is any forbidden food corresponding to the target identity information, then the preset second reminder information is output.

[0134] If the target identity information is the second type of preset identity information, then obtain the contraindicated ingredients and contraindicated content values ​​corresponding to the saved target identity information, obtain the target content value of the contraindicated ingredients contained in the target nutrient, and if the target content value exceeds the contraindicated content value, then output the preset third reminder information.

[0135] In one possible implementation, before outputting a preset second reminder message after determining that the target ingredient is any prohibited ingredient corresponding to the target identity information, the method further includes:

[0136] If a fasting time range is stored for the target identity information, then the stored fasting time range corresponding to the target identity information is obtained, and it is determined whether the time when the target food is taken out is within the fasting time range. If so, the operation of outputting the preset second reminder information is executed.

[0137] In one possible implementation, the method further includes:

[0138] Receive each frame of color image, as well as the depth image acquired simultaneously in the same scene as each frame of color image; determine the target depth image to be acquired when storing or retrieving food based on each frame of color image and the corresponding depth image.

[0139] Based on the actual distance corresponding to each pixel in the target depth image and the pre-saved correspondence between the storage location and the actual distance range, the location where the ingredients are stored is determined.

[0140] The stored ingredient information is managed based on the target ingredient name, the target ingredient quantity, the target nutritional components, and the storage location.

[0141] The communication bus mentioned in the above server can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. This communication bus can be divided into address bus, data bus, control bus, etc. For ease of illustration, only one thick line is used to represent it in the diagram, but this does not mean that there is only one bus or one type of bus.

[0142] The communication interface 1002 is used for communication between the above-mentioned electronic device and other devices.

[0143] The memory may include random access memory (RAM) or non-volatile memory (NVM), such as at least one disk storage device. Optionally, the memory may also be at least one storage device located remotely from the aforementioned processor.

[0144] The processors mentioned above can be general-purpose processors, including central processing units, network processors (NPs), etc.; they can also be digital signal processors (DSPs), application-specific integrated circuits, field-programmable gate arrays or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc.

[0145] Based on the above embodiments, this invention also provides a computer-readable storage medium storing a computer program executable by an electronic device. When the program is run on the electronic device, the electronic device performs the following steps:

[0146] Receive the first color image and depth image simultaneously acquired in the same scene;

[0147] The target shoulder key points in the first color image are obtained by using a pre-trained shoulder recognition model.

[0148] Based on the target shoulder key points and the depth image, determine the target height and target shoulder width; based on the number of pixels in each preset depth interval in the depth image, determine the target depth feature vector;

[0149] Based on the correspondence between height, shoulder width, and depth feature vectors and identity information, the target identity information corresponding to the target height, target shoulder width, and target depth feature vectors is determined.

[0150] In one possible implementation, determining the target height and target shoulder width based on the target shoulder key points and the depth image includes:

[0151] Determine the target pixel points corresponding to the key points of the target shoulder in the depth image;

[0152] The target height and target shoulder width are determined based on the actual coordinates of each pixel in the depth image and the target pixel.

[0153] In one possible implementation, the method further includes:

[0154] Face recognition is performed on the first color image. If candidate identity information is identified through the first color image and the candidate identity information is inconsistent with the target identity information, the target identity information is updated using the candidate identity information.

[0155] In one possible implementation, the method further includes:

[0156] If the identity information corresponding to the target height, target shoulder width, and target depth feature vector is not saved, and the face recognition of the first color image fails to identify the candidate identity information, then a preset first reminder message will be output.

[0157] In one possible implementation, the method further includes:

[0158] Obtain a second color image; using a pre-trained food recognition model, obtain the name and quantity of the target food in the second color image;

[0159] Based on the pre-saved correspondence between food names and nutrients, determine the nutritional components corresponding to the target food name; based on the quantity of the target food and the nutritional components, determine the target nutritional components.

[0160] In one possible implementation, the method further includes:

[0161] If the target identity information is a first type of preset identity information, then the forbidden food corresponding to the saved target identity information is obtained; if the target food is any forbidden food corresponding to the target identity information, then the preset second reminder information is output.

[0162] If the target identity information is the second type of preset identity information, then obtain the contraindicated ingredients and contraindicated content values ​​corresponding to the saved target identity information, obtain the target content value of the contraindicated ingredients contained in the target nutrient, and if the target content value exceeds the contraindicated content value, then output the preset third reminder information.

[0163] In one possible implementation, before outputting a preset second reminder message after determining that the target ingredient is any prohibited ingredient corresponding to the target identity information, the method further includes:

[0164] If a fasting time range is stored for the target identity information, then the stored fasting time range corresponding to the target identity information is obtained, and it is determined whether the time when the target food is taken out is within the fasting time range. If so, the operation of outputting the preset second reminder information is executed.

[0165] In one possible implementation, the method further includes:

[0166] Receive each frame of color image, as well as the depth image acquired simultaneously in the same scene as each frame of color image; determine the target depth image to be acquired when storing or retrieving food based on each frame of color image and the corresponding depth image.

[0167] Based on the actual distance corresponding to each pixel in the target depth image and the pre-saved correspondence between the storage location and the actual distance range, the location where the ingredients are stored is determined.

[0168] The stored ingredient information is managed based on the target ingredient name, the target ingredient quantity, the target nutritional components, and the storage location.

[0169] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0170] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to this application. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0171] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0172] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0173] Obviously, those skilled in the art can make various modifications and variations to this application without departing from the spirit and scope of this application. Therefore, if such modifications and variations fall within the scope of the claims of this application and their equivalents, this application also intends to include such modifications and variations.

Claims

1. An image processing method, characterized in that, The method includes: After confirming that the smart refrigerator has been opened, the system controls an image acquisition device to acquire a first color image and a depth image; wherein, the image acquisition device for acquiring the depth image is installed on the top of the smart refrigerator; the system receives the first color image and the depth image acquired simultaneously in the same scene; the first color image and the depth image are images of the same size. The target shoulder key points in the first color image are obtained by using a pre-trained shoulder recognition model. Based on the target shoulder key points and the depth image, the height of the shoulder and the target shoulder width are determined, and the height of the shoulder represents the target height; wherein, the height of the shoulder is the vertical distance from the center point of the shoulder to the ground, or the vertical distance from any shoulder key point to the ground; the target depth feature vector is determined based on the number of pixels in each preset depth interval in the depth image. Based on the correspondence between height, shoulder width, and depth feature vectors and identity information, the target identity information corresponding to the target height, target shoulder width, and target depth feature vectors is determined. The step of determining the target identity information corresponding to the target height, target shoulder width, and target depth feature vectors based on the correspondence between height, shoulder width, and depth feature vectors and identity information includes: The trained classifier is used to determine the height, shoulder width, and depth feature vectors that match the target height, target shoulder width, and target depth feature vectors. The trained classifier outputs the predicted identity information and confidence score based on the correspondence between the saved height, shoulder width, and depth feature vectors and identity information. If the predicted confidence score is greater than a preset threshold, the identity information is determined to be the target identity information. The method further includes: Obtain a second color image; using a pre-trained food recognition model, obtain the name and quantity of the target food in the second color image; Based on the pre-saved correspondence between food names and nutrients, determine the nutritional components corresponding to the target food name; based on the quantity of the target food and the nutritional components, determine the target nutritional components. If the target identity information is a first type of preset identity information, then the forbidden food corresponding to the saved target identity information is obtained; if the target food is any forbidden food corresponding to the target identity information, then the preset second reminder information is output. If the target identity information is the second type of preset identity information, then obtain the contraindicated ingredients and contraindicated content values ​​corresponding to the saved target identity information, obtain the target content value of the contraindicated ingredients contained in the target nutrient, and if the target content value exceeds the contraindicated content value, then output the preset third reminder information.

2. The method according to claim 1, characterized in that, The step of determining the target height and target shoulder width based on the target shoulder key points and the depth image includes: Determine the target pixel points corresponding to the key points of the target shoulder in the depth image; The target height and target shoulder width are determined based on the actual coordinates of each pixel in the depth image and the target pixel.

3. The method according to claim 1, characterized in that, The method further includes: Face recognition is performed on the first color image. If candidate identity information is identified through the first color image and the candidate identity information is inconsistent with the target identity information, the target identity information is updated using the candidate identity information.

4. The method according to claim 3, characterized in that, The method further includes: If the identity information corresponding to the target height, target shoulder width, and target depth feature vector is not saved, and the face recognition of the first color image fails to identify the candidate identity information, then a preset first reminder message will be output.

5. The method according to claim 1, characterized in that, If, after determining that the target ingredient is any prohibited ingredient corresponding to the target identity information, before outputting the preset second reminder information, the method further includes: If a fasting time range is stored for the target identity information, then the stored fasting time range corresponding to the target identity information is obtained, and it is determined whether the time when the target food is taken out is within the fasting time range. If so, the operation of outputting the preset second reminder information is executed.

6. The method according to claim 1, characterized in that, The method further includes: Receive each frame of color image, as well as the depth image acquired simultaneously in the same scene as each frame of color image; determine the target depth image to be acquired when storing or retrieving food based on each frame of color image and the corresponding depth image. Based on the actual distance corresponding to each pixel in the target depth image and the pre-saved correspondence between the storage location and the actual distance range, the location where the ingredients are stored is determined. The stored ingredient information is managed based on the target ingredient name, the target ingredient quantity, the target nutritional components, and the storage location.

7. An electronic device, characterized in that, The electronic device includes at least a processor and a memory, the processor being configured to execute a computer program stored in the memory to implement the steps of the image processing method as described in any of claims 1-6 above.

8. A computer-readable storage medium, characterized in that, It stores a computer program that, when executed by a processor, implements the steps of the image processing method as described in any one of claims 1-6 above.

Images