A food nutrition assessment method based on adaptive fusion and feature enhancement

Abstract

The present invention relates to the technical field of food analysis, and provides a food nutrition assessment method based on adaptive fusion and feature enhancement, including: extracting multi-scale features from a target image of a target object to be assessed, obtaining multi-scale features of the target image, wherein the target image includes an RGB image and a depth image; determining detail enhancement features corresponding to the target scale features according to the target scale features and the scale features with a scale greater than the target scale features in the multi-scale features, wherein the target scale features are features of any scale in the multi-scale features; fusing the detail enhancement features of different target images to obtain multi-scale fusion features; determining semantic enhancement features corresponding to the fusion features of the target scale according to the fusion features of the target scale and the fusion features of a smaller scale; and determining the numerical value of the mapped nutrient according to the semantic enhancement features. This is to solve the problem of low accuracy of the method for food nutrition assessment based on images in the related art.

Description

Technical Field

[0001] The invention relates to the technical field of food analysis, and in particular to a food nutrition assessment method based on adaptive fusion and feature enhancement. Background Art

[0002] Predicting the content of nutrients in food (nutritional assessment) is an important part of food analysis. Nutritional assessment is applicable to various health-related applications, such as food safety management and healthy diet recommendations. Traditional nutritional assessment techniques often analyze the nutrient content of food based on the chemical or physical properties of food, and most of them are used for scientific research and require professional experimental equipment, making them difficult to popularize.

[0003] With the development of deep learning, visual food analysis has emerged. It has been well applied in various food analysis fields (food identification, ingredient prediction, recipe retrieval). Deep learning technology can analyze food images quickly and non-destructively, and a large number of deep learning nutritional assessment technologies have emerged.

[0004] The deep learning method in the related technology directly applies the deep learning method to nutrition assessment without considering feature extraction and feature processing in a targeted manner, resulting in low accuracy when making nutrition assessment predictions based on general image feature extraction methods.

[0005] The image-based food nutrition assessment method in the related art has the problem of low accuracy, and no effective solution has been proposed yet. Summary of the invention

[0006] The invention provides a method for food nutrition assessment based on adaptive fusion and feature enhancement, which at least solves the problem of low accuracy of the method for food nutrition assessment based on images in the related art.

[0007] The invention provides a food nutrition assessment method based on adaptive fusion and feature enhancement, comprising: performing multi-scale feature extraction on a target image of a target object to be assessed to obtain multi-scale features of the target image, wherein the target image comprises an RGB image and a depth image; determining detail enhancement features corresponding to the target scale features according to target scale features and scale features having a scale greater than the target scale features in the multi-scale features, wherein the target scale features are features of any scale in the multi-scale features; performing feature fusion on detail enhancement features of different target images to obtain multi-scale fusion features; determining semantic enhancement features corresponding to the fusion features of the target scale according to the fusion features of the target scale and the fusion features of a smaller scale; and determining the numerical values ​​of the mapped nutrients according to the semantic enhancement features.

[0008] The detail enhancement features of different target images are feature-fused to obtain multi-scale fusion features, including: obtaining fusion weights of RGB images and depth images, wherein the fusion weights are obtained by adaptive adjustment; according to the fusion weights, weighted summing of the detail enhancement features of the RGB image and the detail enhancement features of the depth image is performed to obtain fusion features; wherein the weighted summing formula is as follows:

[0009] ;

[0010] In the formula, is the fusion feature of the i-th scale; It is the detail enhancement feature of RGB image. The fusion weights of the detail enhancement features of the RGB image, To enhance the details of the depth image, Fusion weights for detail enhancement of depth image features; is a minimum value, used to prevent the denominator from being 0; ⊕ represents element-wise addition.

[0011] The present invention provides a food nutrition assessment method based on adaptive fusion and feature enhancement, which performs multi-scale feature extraction on a target image of a target object to be assessed and obtains multi-scale features of the target image, including: performing multi-scale feature extraction on the target image through a visual encoder to obtain multi-scale features, wherein the size of the multi-scale features is reduced by a factor of 1 / 2, and the number of channels is increased by a factor of 2.

[0012] The invention provides a method for food nutrition assessment based on adaptive fusion and feature enhancement, which determines the detail enhancement feature corresponding to the target scale feature according to the target scale feature and the scale feature whose scale is larger than the target scale feature in the multi-scale feature, including: downsampling multiple scale features whose scale is larger than the target scale feature to obtain multiple corresponding downsampled features, wherein the target scale feature is a feature whose size exceeds a preset size; summing the target scale feature and the multiple downsampled features to obtain the detail enhancement feature corresponding to the target scale feature; wherein downsampling is performed by the following formula:

[0013] ;

[0014] In the formula, express corresponding detail enhancement features; Represents the i-th scale feature of the RGB image; Indicates a dimensionality reduction convolution operation based on a 1×1 convolution kernel. The subscript i and subscript j of represent different 1×1 convolution kernels, which are used to classify the channel dimensions of the i-th scale feature and the j-th scale feature into the same channel dimension; Indicates a downsampling operation, which is used to classify the sizes of features at each scale into the same size. Indicates that a downsampling operation is performed based on the downsampling degree corresponding to the j-th scale.

[0015] The invention provides a food nutrition assessment method based on adaptive fusion and feature enhancement, which determines the semantic enhancement feature corresponding to the fusion feature of the target scale according to the fusion feature of the target scale and the fusion feature of a smaller scale, including: upsampling the fusion feature of the target scale smaller than the target scale to obtain the corresponding upsampled feature, wherein the fusion feature of the target scale is a fusion feature whose size does not exceed a preset size; performing convolution operation and residual connection on the upsampled feature to obtain the corresponding target upsampled feature; summing the fusion feature of the target scale and the target upsampled feature to obtain the semantic enhancement feature corresponding to the fusion feature of the target scale;

[0016] ;

[0017] ;

[0018] In the formula, is the semantic enhancement feature corresponding to the i-th scale feature, is the fusion feature of the i-th scale, is the up-sampled feature of the i-th scale; Upsampling for bilinear interpolation operations; is the convolutional layer; ReLU is the nonlinear activation function.

[0019] The present invention provides a food nutrition assessment method based on adaptive fusion and feature enhancement, which determines the numerical value of the mapped nutrient according to the semantic enhancement feature, including: performing a global average pooling operation on the semantic enhancement feature to obtain a global feature vector; performing a splicing operation on the global feature vector to obtain a comprehensive feature vector; and determining the mapped nutrient numerical value according to the comprehensive feature vector through a fully connected layer, wherein the fully connected layer is used to map the comprehensive feature vector and the predicted nutrient numerical value.

[0020] The invention provides a method for food nutrition assessment based on adaptive fusion and feature enhancement. After determining the value of the mapped nutrient according to the semantic enhancement feature, the method further includes: calculating the loss value using a loss function according to the nutrient value and the true value; optimizing using an optimizer according to the loss value; wherein the loss function is as follows:

[0021] ;

[0022] ;

[0023] In the formula, M is the number of predicted nutrient categories, m is the number of samples trained in the same batch, is the predicted value of the qth nutrient of the pth sample, is the true value, is the loss of qth type of nutrients, is the loss value.

[0024] The present invention provides a food nutrition assessment method based on adaptive fusion and feature enhancement, which performs multi-scale feature extraction on a target image of a target object to be assessed. Before obtaining the multi-scale features of the target image, the method further includes: obtaining multiple groups of training data, wherein each group of training data includes a target image and corresponding nutrient values; training is performed using the training data until the loss value of the loss function reaches a preset loss range, thereby determining that the training is completed.

[0025] An electronic device provided by an embodiment of the present invention includes: a processor and a memory storing a program, wherein the program includes instructions, and when the instructions are executed by the processor, the processor executes any of the above methods.

[0026] The present invention provides a computer program product according to an embodiment of the present invention, including a computer program / instruction. When the computer program / instruction is executed by a processor, any of the above methods is implemented.

[0027] The invention provides a food nutrition assessment method, electronic device and computer program product based on adaptive fusion and feature enhancement, which solves the problem of low accuracy of image-based food nutrition assessment methods in related technologies. By performing detail enhancement, adaptive fusion and semantic information enhancement on image features of different modalities (RGB images, depth images) and different scales, key image features that can fully map nutrients are obtained, thereby obtaining more accurate nutrient values, which significantly improves the accuracy of food nutrition assessment. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the prior art descriptions. Obviously, the drawings described below are only some embodiments of the present invention, and for ordinary technicians in this field, other embodiments can be obtained based on these drawings without creative work.

[0029] Figure 1 It is a step flow chart of a food nutrition assessment method based on adaptive fusion and feature enhancement in an embodiment of the present invention.

[0030] Figure 2It is a schematic diagram of combining the food nutrition assessment method and the deep learning model in an embodiment of the present invention.

[0031] Figure 3 It is a schematic diagram of a detail information enhancement module in an embodiment of the present invention.

[0032] Figure 4 It is a schematic diagram of an adaptive fusion module in an embodiment of the present invention.

[0033] Figure 5 It is a schematic diagram of a semantic information enhancement module in an embodiment of the invention.

[0034] Figure 6 It is a schematic diagram of the structure of an electronic device in an embodiment of the invention. DETAILED DESCRIPTION

[0035] The embodiments of the present invention will be described in more detail below with reference to the accompanying drawings. Although certain embodiments of the present invention are shown in the accompanying drawings, it should be understood that the present invention can be implemented in various forms and should not be construed as being limited to the embodiments described herein, which are instead provided to provide a more thorough and complete understanding of the present invention. It should be understood that the drawings and embodiments of the present invention are only for exemplary purposes and are not intended to limit the scope of protection of the present invention.

[0036] Predicting the content of nutrients in food (nutritional assessment) is an important part of food analysis. Nutritional assessment is applicable to various health-related applications, such as food safety management and healthy diet recommendations. With the development of deep learning, visual food analysis has emerged. It has been well applied in various food analysis fields (food identification, ingredient prediction, recipe retrieval), and deep learning technology can analyze food images quickly and non-destructively. Deep learning nutrition assessment technologies have emerged in large numbers.

[0037] The deep learning method in the related technology directly applies the deep learning method to nutrition assessment without considering feature extraction and feature processing in a targeted manner, resulting in low accuracy when making nutrition assessment predictions based on general image feature extraction methods.

[0038] To do this, see Figure 1 As shown, the present invention provides a method for food nutrition assessment based on adaptive fusion and feature enhancement, including:

[0039] Step S101 , performing multi-scale feature extraction on a target image of a target object to be evaluated to obtain multi-scale features of the target image, wherein the target image includes an RGB image and a depth image.

[0040] Step S102 , determining a detail enhancement feature corresponding to the target scale feature according to the target scale feature and a scale feature whose scale is larger than the target scale feature in the multi-scale feature, wherein the target scale feature is a feature of any scale in the multi-scale feature.

[0041] Step S103, performing feature fusion on detail enhancement features of different target images to obtain multi-scale fusion features.

[0042] Step S104, determining the semantic enhancement feature corresponding to the fusion feature of the target scale according to the fusion feature of the target scale and the fusion feature of a smaller scale.

[0043] Step S105, determining the value of the mapped nutrient according to the semantic enhancement feature.

[0044] The target objects to be evaluated refer to the various foods that require nutritional assessment.

[0045] Multi-scale features refer to the features obtained by analyzing images at different scales (resolutions). By processing images at different scales, the details of the image can be restored and the artifacts and noise generated during the sharpening process can be reduced.

[0046] According to the target scale feature in the multi-scale feature and the scale feature in the multi-scale feature whose scale is larger than the scale of the target scale feature, the detail enhancement feature corresponding to the target scale feature can be determined. The scale feature whose scale is larger than the scale of the target scale feature means that the size of the scale feature is larger than the size of the target scale feature, and the level of the scale feature is smaller than the level of the target scale feature.

[0047] According to the fusion features of the target scale and the fusion features of a scale smaller than the target scale, the semantic enhancement features corresponding to the fusion features of the target scale are determined. The fusion features of a scale smaller than the target scale refer to the fusion features having a size smaller than the fusion features of the target scale, and the level of the fusion features is the level of the fusion features of the target scale plus 1. RGB images are composed of three color channels: red, green, and blue, and each color channel contains the grayscale information of the image. RGB images provide color information and are mainly used for 2D image processing and display. For example, photos and video screenshots seen in daily life.

[0048] Depth images, also called distance images or 3D images, usually have only one channel. Depth images provide distance information and are mainly used for 3D scene understanding and reconstruction. Depth images can be obtained by using structured light, time of flight, binocular vision, and depth sensors. It is understood that the methods for obtaining depth images include but are not limited to these.

[0049] Detail enhancement features can make image features contain more detail information, improve the richness of image features, and help to understand the image content more accurately. For example, the detail information in large-scale features is used to supplement and enhance the detail information of small-scale features. Among them, the level of large-scale features is before the level of small-scale features, and the size of large-scale features is larger than that of small-scale features.

[0050] Multi-scale fusion features can provide a more accurate mapping relationship between nutrients and image features. For example, for meat products, multi-scale fusion features can better reflect the relationship between the texture (related to protein content), color (related to freshness and changes in nutrients), and shape (related to quality) of meat and nutrients, which helps to accurately assess its nutritional content.

[0051] Semantic enhancement features can enhance the semantic information of image features at different scales, so that image features can better reflect the semantic concepts related to nutrition in food images. For example, semantic information in small-scale features is used to supplement and enhance the semantic information of large-scale features.

[0052] The above method provided by the embodiment of the invention performs detail enhancement, adaptive fusion and semantic information enhancement on image features of different modalities (RGB images, depth images) and different scales, thereby obtaining key image features that can fully map nutrients, and then obtaining more accurate nutrient values, thereby significantly improving the accuracy of food nutritional assessment.

[0053] Optionally, in step S101, multi-scale feature extraction is performed on a target image of a target object to be evaluated to obtain multi-scale features of the target image, including:

[0054] Perform data augmentation operations on the RGB image and depth image of the target object, such as random horizontal flipping.

[0055] A feature extractor with the same structure but no shared parameters is used to perform multi-scale feature extraction on the data-enhanced RGB image and depth image respectively.

[0056] It can be understood that performing data augmentation operations helps to more fully obtain the image features of the target object.

[0057] Preferably, in step S101, multi-scale feature extraction is performed on a target image of a target object to be evaluated to obtain multi-scale features of the target image, including:

[0058] Multi-scale features are extracted from the target image through a visual encoder to obtain multi-scale features, wherein the size of the multi-scale features is reduced by a factor of 1 / 2 and the number of channels is increased by a factor of 2.

[0059] RGB images Multi-scale features It can be obtained by the following formula:

[0060] ;

[0061] ;

[0062] In the formula, Represents the visual encoder of the RGB image; i represents the i-th scale, 1≤i≤n; Represents a dimension The real tensor space of ; H and W represent the height and width dimensions respectively, which decrease at a rate of 1 / 2; C represents the number of channels, which increases at a rate of 2.

[0063] It is understandable that the values ​​from 1 to i and then to n are getting larger, but the size from the 1st scale to the i-th scale and then to the n-th scale is getting smaller, and the number of channels from the 1st scale to the i-th scale and then to the n-th scale is getting larger. The "i-th scale" described in the embodiments of the present invention is equivalent to the "i-th layer".

[0064] Depth Image Multi-scale features It can be obtained by the following formula:

[0065] ;

[0066] ;

[0067] In the formula, Visual encoder representing depth images.

[0068] Obviously, since the visual encoder used is structurally the same, the multi-scale features of RGB images and the multi-scale features of depth images are similar in form and acquisition method.

[0069] It is understandable that those skilled in the art can select different size reduction ratios and channel number increase ratios according to actual needs.

[0070] Preferably, in step S102, determining a detail enhancement feature corresponding to the target scale feature according to the target scale feature and a scale feature whose scale is larger than the target scale feature in the multi-scale features includes:

[0071] Step S1021, down-sampling the scale feature whose scale is larger than the target scale feature to obtain a corresponding plurality of down-sampled features, wherein the target scale feature is a feature whose size exceeds a preset size, and the preset size can be determined by those skilled in the art according to actual needs or empirical values.

[0072] Step S1022: sum the target scale feature and the multiple down-sampled features to obtain a detail enhancement feature corresponding to the target scale feature.

[0073] Among them, the multi-scale features of RGB images are expressed by the following formula: Downsampling:

[0074] ;

[0075] In the formula, express corresponding detail enhancement features; Represents the i-th scale feature of the RGB image; Indicates a dimensionality reduction convolution operation based on a 1×1 convolution kernel. The subscript i and subscript j of represent different 1×1 convolution kernels, which are used to classify the channel dimensions of the i-th scale feature and the j-th scale feature into the same channel dimension; Indicates a downsampling operation, which is used to classify the sizes of features at each scale into the same size. Indicates that a downsampling operation is performed based on the downsampling degree corresponding to the j-th scale.

[0076] It can be understood that the above-mentioned i-th scale feature is a target scale feature, the above-mentioned j-th scale feature is a scale feature whose scale is larger than the target scale feature, and j is smaller than i.

[0077] The multi-scale features of the depth image are obtained by Downsampling:

[0078] ;

[0079] In the formula, express corresponding detail enhancement features; Represents the i-th scale feature of the depth image.

[0080] Understandably, , They are the object scale features in RGB images and depth images respectively. The detail enhancement features of RGB images and depth images are similar in form and acquisition method.

[0081] Preferably, in step S103, the detail enhancement features of different target images are fused to obtain multi-scale fusion features, including:

[0082] Step S1031, obtaining the fusion weight of the RGB image and the depth image, wherein the fusion weight is obtained by adaptive adjustment.

[0083] Step S1032: performing weighted summation on the detail enhancement features of the RGB image and the detail enhancement features of the depth image according to the fusion weights to obtain fusion features.

[0084] The weighted summation formula is as follows:

[0085] ;

[0086] In the formula, is the fusion feature of the i-th scale; The fusion weights of the detail enhancement features of the RGB image, Fusion weights for detail enhancement of depth image features; is a minimum value, used to prevent the denominator from being 0; ⊕ represents element-wise addition.

[0087] Preferably, in step S104, determining the semantic enhancement feature corresponding to the fusion feature of the target scale according to the fusion feature of the target scale and the fusion feature of a smaller scale includes:

[0088] Step S1041, up-sample the fused features that are one scale smaller than the target scale to obtain corresponding up-sampled features, wherein the fused features of the target scale are fused features whose size does not exceed a preset size, and the preset size can be determined by those skilled in the art according to actual needs or empirical values.

[0089] Step S1042, performing convolution operation and residual connection on the upsampled features to obtain corresponding target upsampled features.

[0090] Step S1043, summing the fused features of the target scale and the target up-sampled features to obtain semantic enhancement features corresponding to the fused features of the target scale.

[0091] ;

[0092] ;

[0093] In the formula, is the semantic enhancement feature corresponding to the i-th scale feature, is the fusion feature of the i-th scale, is the up-sampled feature of the i-th scale; It is a bilinear interpolation operation, used for upsampling; is the convolutional layer; ReLU is the nonlinear activation function.

[0094] It can be understood that the fusion feature of the i-th scale is the fusion feature of the target scale, and the fusion feature of the i+1-th scale is the fusion feature of a scale smaller than the target scale. Preferably, in step S105, the value of the mapped nutrient is determined according to the semantic enhancement feature, including:

[0095] Step S1051, performing a global average pooling operation on the semantic enhancement features to obtain a global feature vector.

[0096] Step S1052, performing concatenation operation on the global feature vector to obtain a comprehensive feature vector.

[0097] Step S1053, determining the mapped nutrient value according to the comprehensive feature vector through a fully connected layer, wherein the fully connected layer is used to map the comprehensive feature vector and the predicted nutrient value.

[0098] Exemplarily, the above-mentioned fully connected layer is three layers.

[0099] Preferably, after determining the value of the mapped nutrient according to the semantic enhancement feature, the method provided by the embodiment of the present invention further includes:

[0100] The loss function is used to calculate the loss value based on the nutrient value and the true value.

[0101] Optimize using the optimizer based on the loss value.

[0102] The loss function is as follows:

[0103] ;

[0104] ;

[0105] In the formula, M is the number of predicted nutrient categories, m is the number of samples trained in the same batch, is the predicted value of the qth nutrient of the pth sample, is the true value, is the loss of qth type of nutrients, is the loss value.

[0106] The objects that the above optimizer can optimize include but are not limited to: parameters set for multi-scale feature extraction, parameters set for detail enhancement features, parameters set for feature fusion, and parameters set for semantic enhancement features. , It is understandable that those skilled in the art can select a specific optimizer and optimization strategy based on actual needs and experience.

[0107] The above method is combined with the Multi-Task Learning (MTL) architecture to simultaneously learn multiple related tasks in the same machine learning model to improve the model's generalization ability. For example, five learning tasks are set to simultaneously predict five types of nutrients: calories, mass, fat content, carbohydrate content, and protein content.

[0108] Preferably, before step S101, the method provided by the embodiment of the present invention further includes:

[0109] Multiple sets of training data are obtained, wherein each set of training data includes a target image and a corresponding nutrient value.

[0110] The training data is used for training until the loss value of the loss function reaches the preset loss range, and the training is determined to be finished.

[0111] It is understandable that, through pre-training, the deep learning model running based on the above method can have higher prediction accuracy earlier, and can achieve food nutrition assessment with higher accuracy as soon as possible.

[0112] Exemplarily, the present invention also conducts experiments to verify the effect of the above method, as follows:

[0113] See also Figure 2 As shown in FIG. 1 , in the process of using a deep learning model to predict nutrients based on image features to evaluate food nutrition, multi-scale feature extraction, detail enhancement, feature fusion, and semantic enhancement are performed on the target image based on the above method. Among them, detail enhancement is performed through a detail information enhancement module, such as Figure 3 As shown; the detail enhancement features of the RGB image and the depth image are fused through the adaptive fusion module, as shown Figure 4 As shown; semantic enhancement is performed through the semantic information enhancement module, such as Figure 5 shown.

[0114] In the above deep learning model, n is set to 4, and features of four scales are extracted for both RGB images and depth images.

[0115] Will In Set to , corresponding to the five types of nutrients: calories, mass, fat, carbohydrates, and protein.

[0116] The above five types of nutrients were evaluated based on the Nutrition5k dataset.

[0117] PMAE (Percent Mean Absolute Error) is used as an indicator to verify the evaluation results of the above deep learning model, as shown in Table 1. The smaller the PMAE, the smaller the error between the model's predicted value and the true value, and the higher the accuracy of the evaluation result.

[0118] Table 1 PMAE values ​​of nutrient evaluation results

[0119] Heat (PMAE%) Quality (PMAE%) Fat (PMAE%) Carbohydrate (PMAE%) Protein (PMAE%) Average (PMAE%) 14.4 11.7 22.5 21.0 19.4 17.8

[0120] It can be found that the PMAE of the nutrient assessment results of the above-mentioned deep learning model is relatively small, indicating that the food nutrition assessment method based on adaptive fusion and feature enhancement provided by the embodiment of the present invention has a higher accuracy rate.

[0121] The embodiment of the invention also provides an electronic device, comprising: at least one processor; and a memory connected to the at least one processor. The memory stores a computer program that can be executed by the at least one processor, and the computer program is used to enable the electronic device to perform the method provided by the embodiment of the invention when executed by the at least one processor.

[0122] refer to Figure 6 , a block diagram of an electronic device that can be used as a server or client of an embodiment of the invention will now be described, which is an example of hardware devices that can be applied to various aspects of the invention. The electronic device is intended to represent various forms of digital electronic computer devices, such as laptop computers, desktop computers, workbenches, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the invention described and / or required herein.

[0123] like Figure 6 As shown, the electronic device includes a computing unit 601, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 602 or a computer program loaded from a storage unit 608 to a random access memory (RAM) 603. In the RAM 603, various programs and data required for the operation of the electronic device can also be stored. The computing unit 601, the ROM 602, and the RAM 603 are connected to each other via a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.

[0124] Multiple components in the electronic device are connected to the I / O interface 605, including: an input unit 606, an output unit 607, a storage unit 608, and a communication unit 609. The input unit 606 can be any type of device that can input information to the electronic device, and the input unit 606 can receive input digital or character information, and generate key signal input related to user settings and / or function control of the electronic device. The output unit 607 can be any type of device that can present information, and can include but is not limited to a display, a speaker, a video / audio output terminal, a vibrator, and / or a printer. The storage unit 608 can include but is not limited to a disk, an optical disk. The communication unit 609 allows the electronic device to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks, and can include but is not limited to a modem, a network card, an infrared communication device, and / or a wireless communication transceiver, such as a Bluetooth device, a WiFi device, a WiMax device, a cellular communication device, and / or the like.

[0125] The computing unit 601 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 601 include, but are not limited to, a CPU, a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing units, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 601 performs the various methods and processes described above. For example, in some embodiments, the method embodiments created by the present invention may be implemented as a computer program, which is tangibly contained in a machine-readable medium, such as a storage unit 608. In some embodiments, part or all of the computer program may be loaded and / or installed on an electronic device via a ROM 602 and / or a communication unit 609. In some embodiments, the computing unit 601 may be configured to perform the above-described method in any other appropriate manner (e.g., by means of firmware).

[0126] The present invention provides a computer program product according to an embodiment of the present invention, including a computer program / instruction. When the computer program / instruction is executed by a processor, any of the above methods is implemented.

[0127] The computer program for implementing the method of the present invention can be written in any combination of one or more programming languages. These computer programs can be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the computer program, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The computer program can be executed entirely on the machine, partially on the machine, partially on the machine as a stand-alone software package and partially on a remote machine, or entirely on a remote machine or server.

[0128] The embodiments of the present invention also provide a non-transitory machine-readable medium storing a computer program, wherein the computer program, when executed by a processor of a computer, is used to cause the computer to execute the method of the embodiments of the present invention.

[0129] In the context of the present invention, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or apparatus. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable signal medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, or infrared systems, devices, or equipment, or any suitable combination of the foregoing. More specific examples of machine-readable storage media may include electrical connections based on one or more lines, portable computer disks, hard disks, random access memories (RAM), read-only memories (ROM), erasable programmable read-only memories (EPROM or flash memory), optical fibers, portable compact disk read-only memories (CD-ROMs), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0130] It should be noted that the term "including" and its variations used in the embodiments of the present invention are open inclusions, that is, "including but not limited to". The term "based on" means "based at least in part on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one other embodiment"; the term "some embodiments" means "at least some embodiments". The modifications of "one" and "multiple" mentioned in the embodiments of the present invention are illustrative and not restrictive. Those skilled in the art should understand that unless otherwise clearly indicated in the context, it should be understood as "one or more".

[0131] The user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in the embodiments of the present invention are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0132] The various steps described in the method implementation methods provided by the embodiments of the present invention can be performed in different orders and / or in parallel. In addition, the method implementation methods may include additional steps and / or omit the steps shown. The scope of protection of the present invention is not limited in this respect.

[0133] The term "embodiment" in this specification refers to specific features, structures or characteristics described in conjunction with the embodiment that can be included in at least one embodiment of the invention. The appearance of this phrase in various places in the specification does not necessarily mean the same embodiment, nor does it mean that it is mutually exclusive with other embodiments and is independent or optional. The various embodiments in this specification are described in a related manner, and the same and similar parts between the various embodiments refer to each other. In particular, for the device, equipment, and system embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and the relevant parts refer to the partial description of the method embodiment.

[0134] The above-described embodiments only express several implementation methods of the present invention, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of protection. It should be pointed out that, for a person of ordinary skill in the art, several modifications and improvements can be made without departing from the concept of the present invention, and these all belong to the scope of protection of the present invention. Therefore, the scope of protection of the present invention shall be subject to the attached claims.

Claims

1. A food nutrition assessment method based on adaptive fusion and feature enhancement, characterized in that: include: Performing multi-scale feature extraction on a target image of a target object to be evaluated to obtain multi-scale features of the target image, wherein the target image includes an RGB image and a depth image; Determine, according to a target scale feature in a multi-scale feature and a scale feature whose scale is larger than the target scale feature, a detail enhancement feature corresponding to the target scale feature, wherein the target scale feature is a feature of any scale in the multi-scale feature; The detail enhancement features of different target images are fused to obtain multi-scale fusion features; Determining a semantic enhancement feature corresponding to the fusion feature of the target scale according to the fusion feature of the target scale and the fusion feature of a smaller scale; Determining a numerical value of a mapped nutrient according to the semantically enhanced feature; Among them, the detail enhancement features of different target images are fused to obtain multi-scale fusion features, including: Obtaining a fusion weight of the RGB image and the depth image, wherein the fusion weight is obtained by adaptive adjustment; According to the fusion weight, weighted summation is performed on the detail enhancement feature of the RGB image and the detail enhancement feature of the depth image to obtain the fusion feature; The weighted summation formula is as follows: ； In the formula, is the fusion feature of the i-th scale; is the detail enhancement feature of the RGB image, is the fusion weight of the detail enhancement feature of the RGB image, is a detail enhancement feature of the depth image, A fusion weight for detail enhancement features of the depth image; is a minimum value, used to prevent the denominator from being 0; ⊕ represents element-wise addition.

2. The method according to claim 1, characterized in that Perform multi-scale feature extraction on the target image of the target object to be evaluated to obtain multi-scale features of the target image, including: Multi-scale features are extracted from the target image through a visual encoder to obtain the multi-scale features, wherein the size of the multi-scale features is reduced by a factor of 1 / 2, and the number of channels is increased by a factor of 2.

3. The method according to claim 1, characterized in that Determining a detail enhancement feature corresponding to the target scale feature according to a target scale feature and a scale feature whose scale is larger than the target scale feature in the multi-scale features includes: Downsampling a plurality of scale features whose scales are larger than the target scale feature to obtain a plurality of corresponding downsampled features, wherein the target scale feature is a feature whose size exceeds a preset size; Summing the target scale feature and the multiple down-sampled features to obtain a detail enhancement feature corresponding to the target scale feature; Among them, downsampling is performed by the following formula: ； In the formula, express corresponding detail enhancement features; represents the i-th scale feature of the RGB image; Indicates a dimensionality reduction convolution operation based on a 1×1 convolution kernel. The subscript i and subscript j of represent different 1×1 convolution kernels, which are used to classify the channel dimensions of the i-th scale feature and the j-th scale feature into the same channel dimension; Indicates a downsampling operation, which is used to classify the sizes of features at each scale into the same size. Indicates that a downsampling operation is performed based on the downsampling degree corresponding to the j-th scale.

4. The method according to claim 1, characterized in that: Determining semantic enhancement features corresponding to the fusion features of the target scale according to the fusion features of the target scale and the fusion features of a smaller scale, including: Upsampling the fused features at a scale smaller than the target scale to obtain corresponding upsampled features, wherein the fused features at the target scale are fused features whose size does not exceed a preset size; Performing convolution operation and residual connection on the up-sampled features to obtain corresponding target up-sampled features; Summing the fused feature of the target scale and the target upsampled feature to obtain a semantic enhancement feature corresponding to the fused feature of the target scale; ； ； In the formula, is the semantic enhancement feature corresponding to the i-th scale feature, is the fusion feature of the i-th scale, is the up-sampled feature of the i-th scale; Upsampling for bilinear interpolation operations; is the convolutional layer; ReLU is the nonlinear activation function.

5. The method according to claim 1, characterized in that Determining the value of the mapped nutrient according to the semantic enhancement feature includes: Performing a global average pooling operation on the semantic enhancement feature to obtain a global feature vector; Performing a concatenation operation on the global feature vectors to obtain a comprehensive feature vector; The mapped nutrient value is determined according to the comprehensive feature vector through a fully connected layer, wherein the fully connected layer is used to map the comprehensive feature vector and the predicted nutrient value.

6. The method according to any one of claims 1 to 5, characterized in that After determining the value of the mapped nutrient according to the semantic enhancement feature, the method further includes: Calculating a loss value using a loss function according to the numerical value and the true value of the nutrient; Optimizing using an optimizer according to the loss value; The loss function is as follows: ； ； In the formula, M is the number of predicted nutrient categories, m is the number of samples trained in the same batch, is the predicted value of the qth nutrient of the pth sample, is the true value, is the loss of qth type of nutrients, is the loss value.

7. The method according to claim 6, characterized in that Before extracting multi-scale features from a target image of a target object to be evaluated and obtaining multi-scale features of the target image, the method further includes: Acquire multiple sets of training data, wherein each set of training data includes a target image and a corresponding nutrient value; The training is performed using the training data until the loss value of the loss function reaches a preset loss range, and the training is determined to be finished.

8. An electronic device, comprising: A processor and a memory storing a program, wherein the program comprises instructions, which, when executed by the processor, cause the processor to perform the method according to any one of claims 1 to 7.

9. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instructions are executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images