Food nutritional ingredient identification method and device based on video stream

Abstract

The invention relates to the technical field of nutritional ingredient identification, and discloses a food nutritional ingredient identification method and device based on video streaming, and the method comprises the steps: obtaining a plurality of frames of images of target food; the nutritional ingredient content value of a target element of a target frame is obtained from the multiple frames of images, the target frame is any one frame of image in the multiple frames of images, and the target element is any one element in multiple elements in the nutritional ingredient table of the target food; and obtaining a nutritional ingredient identification result of the target food according to the nutritional ingredient content value of the target element of the target frame. Therefore, by collecting the multiple frames of images of the target food, the problems that the efficiency is low due to the fact that a single food picture is manually shot, and the accuracy of the recognition result is affected by the fact that manual recognition is not accurate enough and the single picture can be solved, meanwhile, a large number of different types of food nutritional ingredients can be quickly and accurately recognized, and the recognition efficiency is improved. The method is not limited to specific modes and specific types of nutritional ingredient data identification, the flexibility is high, and the operation speed is high.

Description

Technical Field

[0001] The present invention relates to the technical field of nutrient component recognition, and in particular to a method and device for recognizing nutrient components of food based on video stream. Background Art

[0002] With the development of economy, social progress and the continuous improvement of living standards, people pay more attention to food nutrition and health. Reasonable nutrition is conducive to improving people's health. Therefore, identifying the nutritional components of food is conducive to helping people eat healthily.

[0003] In the related art, one method is to identify the nutritional ingredients of physical objects by taking photos. This method requires the user to align the nutritional ingredient table and then click to take a photo. This method requires manual operation, which is inefficient and may cause the recognition result to be inaccurate because the user has not yet aligned with the nutritional ingredient table. At the same time, this method only relies on a single picture, which will also affect the accuracy of the recognition result. Another method is to identify the text information in the nutritional ingredient table through a text recognition model. Since the text recognition model is trained based on a specific type of nutritional ingredient table and a specific type of text data, the text recognition model will have certain limitations when facing the nutritional ingredient recognition of a large number of different types of food. In addition, the text recognition model recognition method occupies a large amount of system memory, which in turn affects the running speed of the system memory. Summary of the invention

[0004] In view of this, the present invention provides a method and device for identifying food nutritional components based on video streams to solve the problem of being unable to quickly and accurately identify nutritional component information.

[0005] According to a first aspect, an embodiment of the present disclosure provides a method for identifying food nutrients based on a video stream, the method comprising:

[0006] Acquire multiple frames of images of target food;

[0007] Obtaining the nutritional content value of the target element of the target frame from the multiple frames of images, wherein the target frame is any frame of the multiple frames of images, and the target element is any element of the multiple elements in the nutritional composition table of the target food;

[0008] Obtaining a nutrient content recognition result of the target food according to the nutrient content value of the target element of the target frame;

[0009] Wherein, obtaining the nutrient content value of the target element of the target frame includes:

[0010] Identify a detection area of ​​the nutrition facts table from the target frame, and identify an element name and a position index of a target element from the detection area;

[0011] Based on the element name and position index of the target element, select an element unit closest to the element name of the target element from multiple element units after the element name of the target element as the element unit of the target element;

[0012] Get the nutrient content value of the target element according to the element name and element unit of the target element.

[0013] The disclosed embodiment, by collecting multiple frames of images of the target food, can solve the problems of low efficiency caused by manually taking a single picture of the target food, inaccurate recognition results caused by manual alignment with the nutritional ingredient table, and the accuracy of the recognition results affected by a single picture. At the same time, it can also quickly and accurately identify the nutritional components of a large number of different types of food, without the need to use different types of sample data to train a large model, reducing system memory and thereby improving the system operation speed.

[0014] In some optional implementations, obtaining the nutrient content value of the target element of the target frame to identify the nutrient content of the target food further includes:

[0015] Determine whether a jitter parameter of a nutrient content value of a target element of a target frame is not greater than a preset threshold;

[0016] If so, the nutrient content value of the target element of the target frame is used as the nutrient identification result of the target food;

[0017] If not, return to the step of acquiring multiple frames of images of the target food.

[0018] The disclosed embodiment achieves the purpose of maintaining multiple frames of images by determining whether the jitter parameters of the nutrient content value of any element in any frame of the multiple frames are within the allowable jitter threshold, thereby achieving accurate and rapid identification of the nutrient content value of any element in any frame of the multiple frames.

[0019] In some optional implementations, obtaining the nutrient content value of the target element according to the element name and element unit of the target element includes:

[0020] According to the element name and element unit of the target element, a target number is obtained from the recognition area between the two;

[0021] Use the target number as the nutrient content value of the target element.

[0022] The disclosed embodiment locates the element name and element unit of the target element, thereby being able to quickly identify the nutrient content value of the target element.

[0023] In some optional implementations, identifying a detection area of ​​the nutrition ingredient table from the target frame, and identifying an element name and a position index of a target element from the detection area, includes:

[0024] Get the testing area to which the nutrition information table belongs;

[0025] Based on the detection area, get multiple text boxes,

[0026] Based on multiple text boxes, multiple texts are obtained, wherein each text box contains one text;

[0027] Based on the text rotation direction of each text box, merge the text in the text boxes on the same line;

[0028] The texts in different lines after merging are combined into a text string in the same line, and then the element name and position index of the target element are obtained.

[0029] The disclosed embodiment further obtains the nutritional content value of any element in any frame image through the element name and position index of the target element, and can easily add new nutritional elements that need to be identified without the need for additional training of a large amount of sample text data. Compared with traditional text structured recognition models, it can significantly reduce system memory and thus improve recognition speed.

[0030] In some optional implementations, the process of identifying the element unit of the target element includes:

[0031] Determine whether the element unit of the target element is a universal unit;

[0032] If so, use the common unit as the element unit of the target element;

[0033] If not, convert the non-universal unit to the universal unit to obtain the element unit of the target element.

[0034] In the process of identifying the element unit of the target element, the embodiment of the present disclosure further unifies the element unit, which is conducive to meeting the usage habits of the general users.

[0035] In some optional embodiments, the method for identifying food nutrients based on video stream in the embodiments of the present disclosure, in the process of identifying the element name of the target element, identifies the element name of the target element from multiple candidate names in order of priority; in the process of identifying the element unit of the target element, identifies the element unit of the target element from multiple candidate units in order of priority.

[0036] The disclosed embodiment can preferentially search for nutrient element names and units with a high probability of appearing according to the priority order. If they are found, there is no need to continue searching for other candidate names and candidate unit combinations, thereby improving the search speed.

[0037] According to a second aspect, an embodiment of the present disclosure provides a device for identifying food nutrients based on a video stream, the device comprising:

[0038] A multi-frame image acquisition module, used for acquiring multi-frame images of target food;

[0039] A nutrient component acquisition module, used to acquire the nutrient component content value of a target element in a target frame from multiple frames of images of the target food, wherein the target frame is any frame of the multiple frames, and the target element is any element of the multiple elements in the nutrient component table of the target food;

[0040] The recognition result acquisition module is used to obtain the nutrient content recognition result of the target food according to the nutrient content value of the target element of the target frame;

[0041] Nutritional information acquisition module, including:

[0042] An element name recognition submodule, used to recognize a detection area of ​​the nutrition composition table from a target frame, and to recognize an element name and a position index of a target element from the detection area;

[0043] An element unit identification submodule, for selecting, based on the element name and position index of the target element, an element unit closest to the element name of the target element from a plurality of element units following the element name of the target element as the element unit of the target element;

[0044] The element content acquisition submodule is used to obtain the nutrient content value of the target element according to the element name and element unit of the target element.

[0045] According to a third aspect, an embodiment of the present disclosure provides a computer device, including:

[0046] The memory and the processor are communicatively connected to each other, the memory stores computer instructions, and the processor executes the method for identifying food nutrients based on video stream in the first aspect or any embodiment of the first aspect by executing the computer instructions.

[0047] According to the fourth aspect, an embodiment of the present disclosure provides a computer-readable storage medium, on which computer instructions are stored, and the computer instructions are used to enable a computer to execute the video stream-based food nutrient identification method in the first aspect or any embodiment of the first aspect.

[0048] According to a fifth aspect, the present invention provides a computer program product, comprising computer instructions for causing a computer to execute the method for identifying food nutrients based on a video stream according to the first aspect or any corresponding embodiment thereof. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] In order to more clearly illustrate the specific implementation methods of the present invention or the technical solutions in the prior art, the drawings required for use in the specific implementation methods or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some implementation methods of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0050] Figure 1 is a flow chart of a method for identifying food nutrients based on video streams according to an embodiment of the present invention;

[0051] Figure 2 is an image schematic diagram of a vertical table type nutrition ingredient table according to an embodiment of the present invention;

[0052] Figure 3 is an image schematic diagram of a horizontal table type nutrition ingredient table according to an embodiment of the present invention;

[0053] Figure 4 is an image schematic diagram of a text table type nutrition ingredient table according to an embodiment of the present invention;

[0054] Figure 5 is a flow chart of another method for identifying food nutrients based on video streams according to an embodiment of the present invention;

[0055] Figure 6 is a flow chart of another method for identifying food nutrients based on video stream according to an embodiment of the present invention;

[0056] Figure 7 is a schematic diagram of the overall process of a method for identifying food nutrients based on video streams according to an embodiment of the present invention;

[0057] Figure 8 is a structural block diagram of a device for identifying food nutrients based on video stream according to an embodiment of the present invention;

[0058] Fig. 9 It is a schematic diagram of the hardware structure of a computer device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0059] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.

[0060] According to an embodiment of the present invention, an embodiment of a method for identifying food nutrients based on a video stream is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0061] In this embodiment, a method for identifying food nutrients based on video stream is provided, which can be used in mobile terminals such as mobile phones, tablet computers, etc. Figure 1 is a flow chart of a method for identifying food nutrients based on video stream according to an embodiment of the present invention. Figure 1 As shown, the process includes the following steps:

[0062] Step S101, acquiring multiple frames of images of target food.

[0063] Specifically, the target food may be a nutrient or health product or milk powder or beverage in daily life. Multiple frames of images may be collected by a mobile terminal or a camera at preset intervals, wherein the content of any frame of image includes the target food. For example, the preset interval is 0.3s, and the preset interval may be flexibly set in combination with actual conditions. The mobile terminal or camera automatically collects multiple frames of images of the target food based on 0.3s. For example, non-continuous i-th frame, k-th frame, and m-th frame in the video stream are extracted as multiple frames of images of the target food at intervals of 0.3s. For example, continuous i-th frame, j-th frame, and k-th frame may also be extracted as multiple frames of images of the target food at intervals of 0.3s. When the image quality is good, the number of frames of the extracted image may be less, and when the image quality is poor, the number of frames of the extracted image may be more.

[0064] The disclosed embodiment can solve the problems of low efficiency caused by manually taking a single picture of the target food, inaccurate recognition results caused by manual failure to align the nutritional composition table, and the problem that a single picture affects the accuracy of the recognition results by collecting multiple frames of images of the target food. Therefore, obtaining multiple frames of images of the target food can make up for the inaccurate recognition of a single frame.

[0065] Step S102, obtaining the nutrient content value of the target element of the target frame from the multiple frames of the target food, wherein the target frame is any frame of the multiple frames, and the target element is any element of the multiple elements in the nutrient composition table of the target food.

[0066] Specifically, the target frame is any frame of the multiple frames, and for any frame, the nutrient content value of the target element (any element) needs to be obtained. For example, if the target food contains 3 frames, the nutrient content value of any element can be obtained from each of the 3 frames, and the nutrient content value of any element can also be obtained from the first, second, or third frame of the 3 frames. Figure 2 As shown, it is an image containing a nutritional information table. Figure 2 The nutritional composition table in the table includes: TotalFat (total fat), saturated fat (saturated fat), TransFat (trans fat), Cholesterol (cholesterol), Sodium (sodium), TotalCarbohydrate (total carbohydrates), DietaryFiber (dietary fiber), TotalSugars (total sugars), AddedSugars (added sugars), Protein (protein). The target element represents any one of these elements. By automatically taking multiple frames of images, the nutrient content value of any nutrient element in any frame of the image is obtained, that is, the target element is obtained. Figure 2 The value after any element in Figure 2 For example, when it is necessary to identify the nutritional content values ​​of TotalFat (total fat), saturated fat (saturated fat), and Trans Fat (trans fat) in the nutritional composition table, the nutritional content values ​​of the three elements are obtained according to the nutritional content values ​​of the three target elements. For example, for Figure 2 For example, when it is necessary to identify the nutritional content values ​​of all elements in the nutritional table, the nutritional content values ​​of the target elements in each frame image are used to obtain the nutritional identification results of the target food.

[0067] Step S103, obtaining the nutrient content of the target food according to the nutrient content value of the target element of the target frame.

[0068] Specifically, this is equivalent to fusing the nutritional content value of the target element of the target frame (any frame image) to obtain the nutritional recognition result of any element in the nutritional table. For example, the multi-frame image is 3 frames of images, and according to the jitter of the nutritional content value of the target element (any element) of any frame, it is determined whether the nutritional content value of the target element of the frame needs to be taken as the final result. If the jitter of the nutritional content value of the target element (any element) of any frame of the 3 frames is within the allowable range, at the same time, the recognition result of the nutritional content value of the target element (any element) of any frame of the 3 frames is taken as the final nutritional recognition result of the target food. Otherwise, return to step S102 to re-capture the multi-frame image of the target food.

[0069] The disclosed embodiment can randomly collect multiple frames of images of the nutritional composition table of various target foods, such as Figure 2 As shown, it is a vertical table type nutrition information table image, such as Figure 3 As shown, this is a horizontal table type nutrition information table image, such as Figure 4 The figure shows a nutritional ingredient table image of a plain text table type. Therefore, the embodiments of the present disclosure are not limited to specific modes or specific types of nutritional ingredient data recognition. The embodiments of the present disclosure can quickly and accurately recognize the nutritional ingredients of a large number of different types of food without using different types of sample data to train a large model, thereby reducing system memory and improving system operation speed.

[0070] In a specific example, Figure 1 In the above step S102, obtaining the nutrient content value of the target element in the target frame from the multiple frames of the target food image includes:

[0071] Step 1021, identifying a detection area of ​​the nutrition composition table from the target frame, and identifying an element name and a position index of a target element from the detection area.

[0072] Specifically, for example, Figure 2 For example, it can be recognized by OCR image recognition Figure 2 The detection area of ​​the nutrition facts table in the image frame is identified, and the element name and position index of any element (target element) are identified from the detection area where the nutrition facts table is located. For example, the element name can be protein, fat or carbohydrate, and the element index represents Figure 2 Therefore, through the OCR image recognition method, the element names and position indexes of multiple elements can be identified from the detection area of ​​the nutritional composition table.

[0073] Step 1022: Based on the element name and position index of the target element, select an element unit closest to the element name of the target element from a plurality of element units following the element name of the target element as the element unit of the target element.

[0074] Specifically, for example, in Figure 2 In the , based on the element name and position index of the target element (Total Fat), the element unit (g) closest to the element name of the target element (Total Fat) is selected from the multiple element units after the element name of the target element (Total Fat) as the element unit of the target element (Total Fat). According to the statistical analysis of the nutritional composition table, it can also be found that whether it is Figure 2 or Figure 3 or Figure 4 All types of tables in the table support a principle, that is, the element name and the corresponding element unit meet the shortest distance, and there will be no other nutrient elements in the text between the two. The shortest distance is defined according to the order of human reading direction, starting from the upper left and reading to the right. If there is a line break, it will wrap to the leftmost of the next line, and so on, and finally read to the lower right of the text. Therefore, no matter what type of table it is, no matter how many rows there are, the embodiments of the present disclosure can splice these texts into one row according to the definition of the reading order to achieve a one-dimensional text string, which is convenient for calculating the distance between texts.

[0075] Step S1023, obtaining the nutrient content value of the target element according to the element name and element unit of the target element.

[0076] In a specific example, the above step S1023, according to the element name and element unit of the target element, obtains the nutrient content value of the target element, including:

[0077] Step a1, obtaining a target number from a recognition area between the element name and the element unit of the target element;

[0078] Step a2, taking the target number as the nutrient content value of the target element.

[0079] Specifically, for example, in Figure 2 In the example, based on the element name and element unit (g) of the target element (Total Fat), the target number (0) between the two is obtained, and then the nutrient content value of the target element is obtained as 0.

[0080] Therefore, the embodiment of the present disclosure can quickly identify the nutrient content value of any element in the nutritional composition table of different types of target food by executing the above steps S1022-S1023. It not only has high recognition efficiency but also high recognition accuracy. In addition, it has high flexibility and fast running speed.

[0081] In this embodiment, a method for identifying food nutrients based on video stream is provided, which can be used in the above-mentioned mobile terminals, such as mobile phones, tablet computers, etc. Figure 5 is a flow chart of a method for identifying food nutrients based on video stream according to an embodiment of the present invention. Figure 5 As shown, the above step S103, obtaining the nutrient content value of the target element of the target frame, and identifying the nutrient content of the target food also includes:

[0082] Step S1031 , determining whether a jitter parameter of a nutrient content value of a target element of a target frame is not greater than a preset threshold.

[0083] Specifically, for example, the multi-frame image is 3-frame image, and it is determined whether the jitter parameter of the nutrient content value of any element in any frame of the 3-frame image is not greater than the preset threshold value, in order to determine the stability of the multi-frame image. The preset threshold value is the jitter threshold value of the video image, which is a value set by experience.

[0084] Step S1032: If yes, the nutrient content value of the target element of the target frame is used as the nutrient identification result of the target food.

[0085] Specifically, for example, if the jitter parameters of the nutritional content value of any element in any of the three frames of images do not change within the allowed jitter threshold, the nutritional content value corresponding to any element in any of the three frames of images is obtained and then the jump ends, thereby achieving a fast response and high accuracy. In specific operations, if the mobile phone is pointed at the nutritional composition table of multiple frames of images, the nutritional content value of the target element in the target frame can be identified in about 1-2 seconds.

[0086] Step S1033: If not, return to step S101 of acquiring multiple frames of images of the target food.

[0087] Specifically, for example, if the jitter parameter of the nutrient content value of any element in any of the above three frames of images is not within the allowable jitter threshold, multiple frames of images of the target food are recaptured, that is, returning to the above step S101.

[0088] Therefore, the embodiment of the present disclosure can accurately and quickly identify the nutrient content value of any element in any frame image by executing the above steps S1031 to S1033.

[0089] In this embodiment, a method for identifying food nutrients based on video stream is provided, which can be used in the above-mentioned mobile terminals, such as mobile phones, tablet computers, etc. Figure 6 is a flow chart of a method for identifying food nutrients based on video stream according to an embodiment of the present invention. Figure 6 As shown, in the above step S1022, the detection area of ​​the nutrition composition table is identified from the target frame, and the element name and position index of the target element are identified from the detection area. The process includes the following steps:

[0090] Step S601, obtaining the detection area to which the nutrition ingredient table belongs.

[0091] Specifically, for Figure 2 For the nutritional information table, the testing area is the area where the vertical table is located. Figure 3 For the nutritional information table, the testing area is the area where the horizontal table is located. Figure 4 For example, the detection area to which the nutrition information table belongs is the area where the text table is located.

[0092] Step S602, obtaining multiple text boxes based on the detection area.

[0093] Specifically, for example, in Figure 2 In the detection area where the nutrition information table is located, the following text contents are included: Total Fat, saturated fat, Trans Fat, Cholesterol, Sodium, Total Carbohydrate, Dietary Fiber, Total Sugars, Added Sugars, Protein, then multiple text boxes are obtained.

[0094] Step S603, obtaining multiple texts based on multiple text boxes, wherein each text box contains one text.

[0095] Specifically, since there is one text in each text box, multiple texts are acquired based on multiple text boxes.

[0096] Step S604: based on the text rotation direction of each text box, merge the texts in the text boxes in the same row.

[0097] Specifically, based on the text rotation direction of each text box, the text is rotated to the right (for some horizontal tables, there may be multiple frames of video images shot in horizontal mode. At this time, the text in the image is rotated 90 degrees clockwise / counterclockwise. On this basis, the text is rotated to the right. In the specific detection process, because the text box has coordinate information, it can be determined whether each text box is directly in the same row. If the height overlap of two detection boxes is greater than 60%, it is considered that the two text boxes are in the same row. In this way, the text in the text boxes in the same row is merged.

[0098] Step S605, merging the texts in different lines into a text string in the same line, and then obtaining the element name and position index of the target element.

[0099] Specifically, the above steps S601 to S605 are equivalent to a mobile phone or camera identifying text, text boxes, merging text lines, and then merging multiple lines of text through OCR recognition. Fuzzy matching + regular expression matching is performed in the text string according to the predefined element name and element unit, allowing slight OCR misrecognition, 20% error in text length, 2 characters can be wrong for an element name of 10 characters, and some special rule matching. Some nutritional elements require the first and last characters to be correct, and the middle characters can have a 20% error (for example, protein is recognized as protain, because fuzzy matching is allowed, so protain will be identified as protein).

[0100] Therefore, the embodiment of the present disclosure identifies the element name and position index of the target element through OCR recognition, and further obtains the nutritional content value of any element in any frame image. It can easily add new nutritional elements that need to be identified without the need for additional training of a large amount of sample text data. Compared with traditional text structured recognition models, it can significantly reduce system memory and thus improve recognition speed.

[0101] In some optional implementations, the method for identifying food nutrients based on video stream in the embodiment of the present disclosure, in the process of identifying the element unit of the target element, includes:

[0102] Step b1, determining whether the element unit of the target element is a universal unit.

[0103] Specifically, for example, the universal unit is a unit of g / mg / mcg, and it is determined whether the element unit of the target element is a universal unit to determine whether it conforms to the usage habits of the general users.

[0104] Step b2: If yes, use the universal unit as the element unit of the target element.

[0105] Step b3: if not, convert the non-universal unit into a universal unit, thereby obtaining the element unit of the target element.

[0106] Specifically, for example, the non-universal unit is generally %, which generally represents the percentage of the DV value (fixed value). The DV value of the element is multiplied by the identified percentage to convert it into the unit of g / mg / mcg.

[0107] Therefore, in the process of identifying the element unit of the target element, the embodiment of the present disclosure further unifies the element unit, which is conducive to meeting the usage habits of the general users.

[0108] In some optional embodiments, the method for identifying food nutrients based on video stream in the embodiments of the present disclosure, in the process of identifying the element name of the target element, identifies the element name of the target element from multiple candidate names in order of priority; in the process of identifying the element unit of the target element, identifies the element unit of the target element from multiple candidate units in order of priority.

[0109] Specifically, the element name and element unit of the target element in the embodiment of the present disclosure are pre-generated based on multiple candidate objects, and these multiple candidate objects are sorted based on priority order. For example: the name of protein contains multiple names, which are sorted in order of priority as A, B, and C. In the actual process of identifying the element name of the target element, if the identification of A fails, B and C are used as backup candidate identification names. Similarly, in the actual process of identifying the element unit of the target element, for example, the macromolecular component is in g, the small molecule is in mg, and the smaller one is in mcg, and there will be a unit of %, in order to prevent the unit from being missed, the macromolecular nutrient element is sorted in order of priority, respectively g, mg, mcg, %, and the small molecule nutrient element is also sorted in order of priority, respectively, mg, mcg, %, g, ​​and so on; the advantage of priority is that the name and unit of the nutrient element with a high probability of occurrence can be searched first, and if it is searched, there is no need to continue searching for other candidate names and candidate unit combinations, thereby improving the search speed. The element names and element units in the embodiments of the present disclosure are pre-defined based on text, so the judgment speed is extremely fast, and if you want to add new identification elements, you only need to offline count the distribution of the element names and element units of the nutrient components.

[0110] like Figure 7 FIG. 1 is a schematic diagram of the overall process of the method for identifying food nutrients based on video streams in an embodiment of the present disclosure. Figure 7In the process, a frame of image is acquired at a preset interval of 0.3s, and then the text content and text detection frame of any frame are identified by OCR extraction, and then the text rotation direction is determined according to the text detection frame information, and then the text is rotated according to the text rotation direction, and then the text lines are merged according to the text detection frame (the text in one line is merged into a text string), and then multiple lines of characters are merged into one line of text string according to the detection frame information of the text string (multiple lines are converted to one line), and then the position of the nutrient name in the matching text string is searched, and then the matching candidate units are searched between the adjacent nutrient elements before the last nutrient element, and then the numbers between the nutrient elements and the units (nutrient element content values) are obtained, and if there is a non-universal unit %, it will be converted into g, mg, mcg according to the fixed value DV, and then each pair of element name, content value, and unit is identified, and finally the multi-frame results are maintained. If the jitter parameter of the identified element content value is within the allowable jitter threshold, the jump result is also, otherwise, return to the step of re-collecting multiple frames of images.

[0111] In the present embodiment, a food nutrient component identification device based on video stream is also provided, and the device is used to realize the above-mentioned embodiment and preferred implementation mode, and the description that has been made is not repeated. As used below, the term "module" can realize the combination of software and / or hardware of predetermined function. Although the device described in the following embodiments is preferably realized with software, the realization of hardware, or the combination of software and hardware is also possible and conceived.

[0112] This embodiment provides a device for identifying food nutrients based on video streams. Figure 8 As shown, including:

[0113] A multi-frame image acquisition module 801 is used to acquire multiple frames of images of target food;

[0114] The nutrient component acquisition module 802 is used to acquire the nutrient component content value of the target element of the target frame from the multiple frames of the target food, wherein the target frame is any frame of the multiple frames, and the target element is any element of the multiple elements in the nutrient component table of the target food;

[0115] The recognition result acquisition module 803 obtains the recognition result of the nutrient components of the target food according to the nutrient content value of the target element of the target frame;

[0116] The nutrient component acquisition module 802 includes:

[0117] The element name recognition submodule 8021 is used to recognize the detection area of ​​the nutrition composition table from the target frame, and recognize the element name and position index of the target element from the detection area;

[0118] The element unit identification submodule 8022 is used to select, based on the element name and position index of the target element, the element unit closest to the element name of the target element from the multiple element units following the element name of the target element as the element unit of the target element;

[0119] The element content acquisition submodule 8023 is used to obtain the nutrient content value of the target element according to the element name and element unit of the target element.

[0120] In some optional implementations, the recognition result acquisition module 803 further includes:

[0121] A judging submodule, used to judge whether a jitter parameter of a nutrient content value of a target element of a target frame is not greater than a preset threshold;

[0122] A first determination submodule is used for, if yes, taking the nutrient content value of the target element of the target frame as the nutrient identification result of the target food;

[0123] The second determination submodule is used to return to the step of acquiring multiple frames of images of the target food if no.

[0124] In some optional embodiments, the nutrient component acquisition module 802 includes:

[0125] The content numerical value recognition submodule is used to obtain the target number from the recognition area between the element name and the element unit of the target element;

[0126] The content value determination submodule is used to use the target number as the nutrient content value of the target element.

[0127] In some optional implementations, the element name identification submodule 8021 includes:

[0128] A first acquisition unit is used to acquire the detection area to which the nutrition composition table belongs;

[0129] The second acquisition unit is used to acquire multiple text boxes based on the detection area.

[0130] A third acquisition unit is used to acquire multiple texts based on multiple text boxes, wherein each text box contains one text;

[0131] A text merging unit, used for merging texts in text boxes of the same line based on the text rotation direction of each text box;

[0132] The text combination unit is used to combine the texts in different lines into a text string in the same line, and then obtain the element name and position index of the target element.

[0133] In some optional implementations, the element unit identification submodule 8022 includes:

[0134] A universal unit determination submodule is used to determine whether the element unit of the target element is a universal unit;

[0135] an element unit determination submodule, for, if yes, using the universal unit as the element unit of the target element;

[0136] The element unit conversion submodule is used to convert the non-universal unit into the universal unit, so as to obtain the element unit of the target element.

[0137] In some optional implementations, in the food nutrient component identification device based on video stream in the embodiment of the present disclosure, the element unit identification submodule 8022 includes an element unit identification unit, and the element name identification submodule 8021 includes an element name identification unit.

[0138] Among them, the element name recognition unit is used to identify the element name of the target element from multiple candidate names in order of priority during the process of identifying the element name of the target element; the element name recognition unit is used to identify the element unit of the target element from multiple candidate units in order of priority during the process of identifying the element unit of the target element.

[0139] The further functional description of each of the above modules and units is the same as that of the above corresponding embodiments and will not be repeated here.

[0140] The video stream-based food nutrient identification device in this embodiment is presented in the form of a functional unit, where the unit refers to an ASIC (Application Specific Integrated Circuit) circuit, a processor and memory that executes one or more software or fixed programs, and / or other devices that can provide the above functions.

[0141] The embodiment of the present invention further provides a computer device having the above-mentioned device for identifying food nutrients based on video stream.

[0142] See also Fig. 9 , Fig. 9 is a schematic diagram of the structure of a computer device provided by an optional embodiment of the present invention, such as Fig. 9As shown, the computer device includes: one or more processors 10, a memory 20, and interfaces for connecting various components, including high-speed interfaces and low-speed interfaces. Various components are connected to each other using different buses for communication, and can be installed on a common mainboard or installed in other ways as needed. The processor can process the instructions executed in the computer device, including instructions stored in or on the memory to display the graphical information of the GUI on an external input / output device (such as, a display device coupled to the interface). In some optional embodiments, if necessary, multiple processors and / or multiple buses can be used together with multiple memories and multiple memories. Similarly, multiple computer devices can be connected, and each device provides some necessary operations (for example, as a server array, a group of blade servers, or a multi-processor system). Fig. 9 A processor 10 is taken as an example.

[0143] The processor 10 may be a central processing unit, a network processor or a combination thereof. The processor 10 may further include a hardware chip. The hardware chip may be a dedicated integrated circuit, a programmable logic device or a combination thereof. The programmable logic device may be a complex programmable logic device, a field programmable gate array, a general purpose array logic or any combination thereof.

[0144] The memory 20 stores instructions executable by at least one processor 10, so that the at least one processor 10 executes the method shown in the above embodiment.

[0145] The memory 20 may include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function; the data storage area may store data created according to the use of the computer device, etc. In addition, the memory 20 may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some optional embodiments, the memory 20 may optionally include a memory remotely arranged relative to the processor 10, and these remote memories may be connected to the computer device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0146] The memory 20 may include a volatile memory, such as a random access memory; the memory may also include a non-volatile memory, such as a flash memory, a hard disk or a solid state drive; the memory 20 may also include a combination of the above types of memory.

[0147] The computer device further comprises a communication interface 30 for the computer device to communicate with other devices or a communication network.

[0148] The embodiment of the present invention also provides a computer-readable storage medium. The method according to the embodiment of the present invention can be implemented in hardware, firmware, or can be implemented as a computer code that can be recorded in a storage medium, or can be implemented as a computer code that is originally stored in a remote storage medium or a non-temporary machine-readable storage medium and will be stored in a local storage medium through a network download, so that the method described herein can be stored in such software processing on a storage medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware. Among them, the storage medium can be a magnetic disk, an optical disk, a read-only storage memory, a random access memory, a flash memory, a hard disk or a solid-state hard disk, etc.; further, the storage medium can also include a combination of the above types of memories. It can be understood that a computer, a processor, a microprocessor controller, or programmable hardware includes a storage component that can store or receive software or computer code. When the software or computer code is accessed and executed by a computer, a processor, or hardware, the method shown in the above embodiment is implemented.

[0149] A part of the present invention may be applied as a computer program product, such as a computer program instruction, which, when executed by a computer, can call or provide the method and / or technical solution according to the present invention through the operation of the computer. Those skilled in the art should understand that the existence of the computer program instruction in a computer-readable medium includes, but is not limited to, a source file, an executable file, an installation package file, etc., and accordingly, the way in which the computer program instruction is executed by the computer includes, but is not limited to: the computer directly executes the instruction, or the computer compiles the instruction and then executes the corresponding compiled program, or the computer reads and executes the instruction, or the computer reads and installs the instruction and then executes the corresponding installed program. Here, the computer-readable medium may be any available computer-readable storage medium or communication medium accessible to the computer.

[0150] Although the embodiments of the present invention have been described in conjunction with the accompanying drawings, those skilled in the art may make various modifications and variations without departing from the spirit and scope of the present invention, and such modifications and variations are all within the scope defined by the appended claims.

Claims

1. A method for identifying food nutrients based on video stream, characterized in that: The method comprises: Acquire multiple frames of images of target food; Obtaining the nutritional content value of the target element in the target frame from the multiple frames of images, wherein the target frame is any frame of the multiple frames of images, and the target element is any one of the multiple elements in the nutritional composition table of the target food; Obtaining a nutrient content recognition result of the target food according to the nutrient content value of the target element of the target frame; Wherein, the step of obtaining the nutrient content value of the target element of the target frame includes: Identifying a detection area of ​​the nutritional composition table from the target frame, and identifying an element name and a position index of the target element from the detection area; Based on the element name and position index of the target element, selecting an element unit closest to the element name of the target element from a plurality of element units following the element name of the target element as the element unit of the target element; According to the element name and element unit of the target element, the nutrient content value of the target element is obtained.

2. The method according to claim 1, characterized in that: Obtaining the nutrient content value of the target element of the target frame according to the nutrient content value of the target food, further comprising: Determine whether a jitter parameter of a nutrient content value of a target element of the target frame is not greater than a preset threshold; If yes, taking the nutritional content value of the target element of the target frame as the nutritional content identification result of the target food; If not, return to the step of acquiring multiple frames of images of the target food.

3. The method according to claim 1, characterized in that According to the element name and element unit of the target element, the nutrient content value of the target element is obtained, including: According to the element name and element unit of the target element, a target number is obtained from the identification area between the two; The target number is used as the nutrient content value of the target element.

4. The method according to claim 1, characterized in that: Identifying a detection area of ​​the nutritional composition table from the target frame, and identifying an element name and a position index of the target element from the detection area, including: Obtaining the detection area to which the nutritional composition table belongs; Based on the detection area, multiple text boxes are obtained, Based on the multiple text boxes, obtain multiple texts, wherein there is one text in each text box; Based on the text rotation direction of each text box, merge the texts in the text boxes in the same row; The texts in different lines after merging are combined into a text string in the same line, thereby obtaining the element name and position index of the target element.

5. The method according to claim 1, characterized in that The process of identifying the element unit of the target element includes: Determine whether the element unit of the target element is a universal unit; If so, use the universal unit as the element unit of the target element; If not, the non-universal unit is converted into the universal unit, thereby obtaining the element unit of the target element.

6. The method according to any one of claims 1 to 5, characterized in that In the process of identifying the element name of the target element, the element name of the target element is identified from multiple candidate names in order of priority; in the process of identifying the element unit of the target element, the element unit of the target element is identified from multiple candidate units in order of priority.

7. A device for identifying food nutrients based on video stream, characterized in that: The device comprises: A multi-frame image acquisition module, used to acquire multiple frames of images of target food; A nutrient component acquisition module, used to acquire a nutrient component content value of a target element in a target frame from multiple frames of images of a target food, wherein the target frame is any frame of the multiple frames, and the target element is any element of multiple elements in a nutrient component table of the target food; An identification result acquisition module, used to obtain the nutritional component identification result of the target food according to the nutritional component content value of the target element of the target frame; The nutrient component acquisition module comprises: An element name recognition submodule, used to recognize a detection area of ​​the nutrition composition table from the target frame, and to recognize an element name and a position index of the target element from the detection area; An element unit identification submodule, configured to select, based on the element name and position index of the target element, an element unit closest to the element name of the target element from a plurality of element units following the element name of the target element as the element unit of the target element; The element content acquisition submodule is used to obtain the nutritional content value of the target element according to the element name and element unit of the target element.

8. A computer device, characterized in that: include: A memory and a processor, wherein the memory and the processor are communicatively connected to each other, the memory stores computer instructions, and the processor executes the method for identifying food nutrients based on video streams as described in any one of claims 1 to 6 by executing the computer instructions.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a computer to execute the method for identifying food nutrients based on video streams according to any one of claims 1 to 6.

10. A computer program product, characterized in that The method comprises computer instructions for causing a computer to execute the method for identifying food nutrients based on video stream according to any one of claims 1 to 6.

Images