Food image recognition method and device and computer readable storage medium

Abstract

The invention provides a food image recognition method and device and a computer readable storage medium. The method comprises the steps of obtaining a to-be-recognized food image, and performing feature extraction processing on the to-be-recognized food image to obtain a shared feature map; the identification difficulty and task identification type of the to-be-identified food image are determined based on the shared feature map, a difficulty score scalar and a task correlation vector are obtained, a feature identification model corresponding to the to-be-identified food image is determined according to the difficulty score scalar, and the difficulty score scalar is related to computing resources required for identifying the to-be-identified food image; and inputting the shared feature map into a feature recognition model corresponding to the to-be-recognized food image to obtain a recognition feature vector, and determining a recognition result of the to-be-recognized food image according to the task correlation vector and the recognition feature vector. The problem that an existing food image recognition scheme is low in recognition efficiency is solved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of food image recognition, in particular to a food image recognition method and device, a computer readable storage medium and an electronic device. BACKGROUND

[0002] Existing food recognition systems usually adopt fixed-structure deep neural network models (such as single ResNet, EfficientNet or multi-task learning models). These models use the same network depth, width and calculation path when processing all input images, regardless of whether their content is simple (such as an apple) or complex (such as a plate of salad containing multiple ingredients).

[0003] This "one-size-fits-all" paradigm has significant drawbacks:

[0004] Waste of resources: for simple images, the complete model calculation is far more than needed, causing waste of computing resources and time, which is not conducive to deployment in mobile devices or edge computing scenarios.

[0005] Performance bottleneck: to ensure the recognition accuracy of complex images, the model must be designed to be large and complex enough. This results in a huge number of model parameters and high inference delay, and may introduce unnecessary noise when processing simple images due to overfitting.

[0006] Static architecture: traditional multi-task learning (such as food classification, calorie estimation, and ingredient segmentation) activates all task branches fixedly, and cannot adaptively select the most relevant task for calculation according to the image content, resulting in low efficiency. SUMMARY

[0007] The main purpose of the present application is to provide a food image recognition method, device, computer readable storage medium and electronic device to at least solve the problem of low recognition efficiency in existing food image recognition solutions.

[0008] To achieve the above purpose, according to one aspect of the present application, a food image recognition method is provided, comprising: acquiring a food image to be recognized, performing feature extraction processing on the food image to be recognized to obtain a shared feature map; determining the recognition difficulty and task recognition type of the food image to be recognized based on the shared feature map to obtain a difficulty score scalar and a task relevance vector, and determining a feature recognition model corresponding to the food image to be recognized according to the difficulty score scalar, wherein the difficulty score scalar is related to the computing resources required to recognize the food image to be recognized; inputting the shared feature map into the feature recognition model corresponding to the food image to be recognized to obtain a recognition feature vector, and determining the recognition result of the food image to be recognized according to the task relevance vector and the recognition feature vector.

[0009] Optionally, determining, based on the shared feature map, a difficulty score scalar and a task relevance vector of the food image to be recognized, comprises: constructing a difficulty perception model, wherein the difficulty perception model is composed of a global average pooling layer and a multi-layer perception; inputting the shared feature map into the difficulty perception model to obtain the difficulty score scalar and the task relevance vector.

[0010] Optionally, determining, according to the difficulty score scalar, a feature recognition model corresponding to the food image to be recognized, comprises: constructing a feature recognition model sequence, wherein the feature recognition model sequence is composed of a plurality of feature recognition models arranged from low to high according to model network layers; constructing a model matching table, wherein the model matching table is a mapping relationship table of the model network layers of the feature recognition model and the difficulty score scalar; selecting, according to the difficulty score scalar and the model matching table, the feature recognition model corresponding to the food image to be recognized from the feature recognition model sequence.

[0011] Optionally, determining, according to the task relevance vector and the recognition feature vector, a recognition result of the food image to be recognized, comprises: determining, according to the task relevance vector, the task recognition type of the food image to be recognized; selecting, based on the task recognition type, a task calculation model corresponding to the food image to be recognized, and inputting the recognition feature vector into the task calculation model to obtain the recognition result corresponding to the food image to be recognized.

[0012] Optionally, determining, according to the task relevance vector, the task recognition type of the food image to be recognized, comprises: inputting the task relevance vector into a Gumbel-Softmax-based sampler to obtain a task activation mask, and determining, according to the task activation mask, the task recognition type corresponding to the food image to be recognized.

[0013] Optionally, the multi-layer perception in the difficulty perception model comprises a hidden layer and an output layer of a preset dimension, wherein the hidden layer adopts a ReLU activation function, the output layer comprises a first output layer and a second output layer, the first output layer is used for outputting a difficulty score scalar, the first output layer adopts a Sigmoid activation function, the second output layer is used for outputting a task relevance vector, and the second output layer adopts a Softmax activation function.

[0014] Optionally, the food image to be identified is subjected to feature extraction processing to obtain a shared feature map, including: constructing a lightweight feature extraction model, inputting the food image to be identified into the lightweight feature extraction model to obtain the shared feature map, wherein the lightweight feature extraction model is a lightweight convolutional neural network model.

[0015] According to another aspect of the present application, a food image recognition device is provided, including: an acquisition unit configured to acquire a food image to be identified, and subject the food image to be identified to feature extraction processing to obtain a shared feature map; a first determination unit configured to determine an identification difficulty and a task identification type of the food image to be identified based on the shared feature map, to obtain a difficulty score scalar and a task relevance vector, and determine a feature recognition model corresponding to the food image to be identified according to the difficulty score scalar; and a second determination unit configured to input the shared feature map into the feature recognition model corresponding to the food image to be identified, to obtain an identification feature vector, and determine an identification result of the food image to be identified according to the task relevance vector and the identification feature vector.

[0016] According to still another aspect of the present application, a computer readable storage medium is provided, including a stored program, wherein the program, when executed, controls a device where the computer readable storage medium is located to perform any of the food image recognition methods.

[0017] According to yet another aspect of the present application, an electronic device is provided, including: one or more processors, a memory, and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, and the one or more programs include instructions for performing any of the food image recognition methods.

[0018] According to the technical solution of the present application, the primary features (shared feature map) of the input food image are extracted, and then the processing difficulty of the image is evaluated, and the task type most suitable for the current image recognition requirement is identified, and the evaluation result determines the degree of calculation resources required for subsequent deep feature extraction and the activation state of the task branch. According to the difficulty score scalar of the image, the system intelligently selects the most suitable deep calculation model, avoiding the waste of resources caused by excessive calculation on simple images. In addition, according to the task relevance vector, the system can accurately activate the task branch that is highly matched with the image task, ensuring that complex images get sufficient calculation resources, while excluding irrelevant task interference, improving the overall recognition accuracy and efficiency. Overall, the dynamic calculation graph and difficulty adaptive recognition scheme proposed in this embodiment realizes the reasonable allocation of calculation resources, improves the comprehensive performance of food image recognition, ensures the recognition accuracy, and significantly reduces the calculation resource consumption, thereby solving the problem of low recognition efficiency of the existing food image recognition scheme. BRIEF DESCRIPTION OF DRAWINGS

[0019] The accompanying drawings, which form a part of the specification, are included to provide a further understanding of the present application, and are incorporated herein by reference. The illustrations are shown to explain the present application and are not intended to limit the present application. In the drawings:

[0020] Figure 1 A hardware structure block diagram of a mobile terminal for performing a food image recognition method according to an embodiment of the present application is shown;

[0021] Figure 2 A flowchart of a food image recognition method according to an embodiment of the present application is shown;

[0022] Figure 3 A flowchart of a specific food image recognition method according to an embodiment of the present application is shown;

[0023] Figure 4 A structure block diagram of a food image recognition device according to an embodiment of the present application is shown. DETAILED DESCRIPTION

[0024] It should be noted that the embodiments and features in the present application can be combined with each other without conflict. The present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0025] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort should fall within the scope of protection of the present application.

[0026] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate for the embodiments of this application described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0027] As described in the background section, existing food image recognition schemes suffer from low recognition efficiency. To address this issue, embodiments of this application provide a food image recognition method, apparatus, computer-readable storage medium, and electronic device.

[0028] The technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention.

[0029] The methods and embodiments provided in this application can be executed on a mobile terminal, computer terminal, or similar computing device. Taking running on a mobile terminal as an example, Figure 1 This is a hardware structure block diagram of a mobile terminal for a food image recognition method according to an embodiment of the present invention. Figure 1 As shown, a mobile terminal may include one or more ( Figure 1 Only one is shown in the diagram. A processor 102 (which may include, but is not limited to, a microprocessor MCU or a programmable logic device FPGA, etc.) and a memory 104 for storing data are also shown. The mobile terminal may further include a transmission device 106 for communication functions and an input / output device 108. Those skilled in the art will understand that... Figure 1 The structure shown is for illustrative purposes only and does not limit the structure of the mobile terminal described above. For example, the mobile terminal may also include components that are more... Figure 1 The more or fewer components shown, or having the same Figure 1 The different configurations shown.

[0030] The memory 104 can be used to store computer programs, such as software programs of application software and modules, such as a computer program corresponding to the food image recognition method in the embodiments of the present application. The processor 102 executes various functional applications and data processing, i.e., implements the above method, by running the computer program stored in the memory 104. The memory 104 can include a high-speed random access memory, and can further include a non-volatile memory, such as one or more magnetic storage devices, flash memories, or other non-volatile solid-state memories. In some examples, the memory 104 can further include a memory remotely arranged with respect to the processor 102, which can be connected to the mobile terminal through a network. Examples of the above network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and a combination thereof. The transmission device 106 is used to receive or send data via a network. The specific examples of the above network can include a wireless network provided by a communication provider of the mobile terminal. In one example, the transmission device 106 includes a network adapter (NIC), which can be connected to other network devices through a base station so as to communicate with the Internet. In one example, the transmission device 106 can be a radio frequency (RF) module, which is used to communicate with the Internet in a wireless manner.

[0031] In the embodiments, a food image recognition method running on a mobile terminal, a computer terminal, or a similar computing device is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0032] Figure 2 is a flowchart of the food image recognition method according to the embodiments of the present application. As shown in Figure 2 , the method includes the following steps:

[0033] In step S201, a food image to be recognized is obtained, and a feature extraction process is performed on the food image to be recognized to obtain a shared feature map;

[0034] In step S202, the difficulty of recognizing the food image to be recognized and the task recognition type are determined based on the shared feature map, a difficulty score scalar and a task relevance vector are obtained, and a feature recognition model corresponding to the food image to be recognized is determined according to the difficulty score scalar, wherein the difficulty score scalar is related to the computing resources required to recognize the food image to be recognized;

[0035] Specifically, the difficulty score scalar is related to the computing resources required to identify the above-mentioned to-be-identified food image, which relates to the number, type, size, position, texture, illumination, and blur degree of the objects in the to-be-identified food image. For example, the difficulty score scalar takes a value of 0-1, 0 represents simple, and 1 represents complex. If the content of the to-be-identified food image is an apple, the difficulty score scalar is 0, indicating that the identification difficulty is simple. If the content of the to-be-identified food image is a plate of salad, the difficulty score scalar is 1, indicating that the identification difficulty is complex.

[0036] In step S203, the shared feature map is input into the feature recognition model corresponding to the to-be-identified food image to obtain an identification feature vector, and the identification result of the to-be-identified food image is determined according to the task relevance vector and the identification feature vector.

[0037] Through the above steps S201, S202, and S203, the primary features (shared feature map) of the input food image are extracted, the processing difficulty of the image is then evaluated, and the task type most suitable for the current image recognition requirement is identified. This evaluation result determines the degree of computing resources required for subsequent deep feature extraction and the activation state of the task branch. According to the difficulty score scalar of the image, the system intelligently selects the most suitable deep computing model, avoiding the waste of resources caused by excessive computation on simple images. In addition, according to the task relevance vector, the system can accurately activate the task branch that is highly matched with the image task, ensuring that complex images receive sufficient computing resources, while excluding irrelevant task interference, thereby improving the overall recognition accuracy and efficiency. Overall, the dynamic computing graph and difficulty adaptive recognition scheme proposed in this embodiment realizes the rational allocation of computing resources, improves the comprehensive performance of food image recognition, ensures the recognition accuracy, significantly reduces the consumption of computing resources, and solves the problem of low recognition efficiency in existing food image recognition schemes.

[0038] In a specific implementation process, the difficulty score scalar and the task relevance vector are determined based on the shared feature map, and the identification difficulty and the task recognition type of the to-be-identified food image are determined, which includes: constructing an identification difficulty perception model, wherein the identification difficulty perception model is composed of a global average pooling layer and a multilayer perception mechanism; inputting the shared feature map into the identification difficulty perception model to obtain the difficulty score scalar and the task relevance vector.

[0039] The method can effectively analyze the primary shared features of the input image by constructing a difficulty perception model composed of a global average pooling layer and a multi-layer perception machine, and then predict the difficulty score and the task relevance vector of the image processing. This design allows the system to intelligently judge the complexity of the image and its relevance to different recognition tasks, dynamically selects the most suitable sub-network for deep feature extraction, and adaptively activates the most relevant task branch for processing.

[0040] Specifically, the output of the difficulty perception model guides the resource allocation controller to select from the pre-set different depth sub-networks, ensuring that complex images receive sufficient computing resources while simple images avoid excessive computation. After the task relevance vector is processed by Gumbel-Softmax sampling, the generated task activation mask further realizes the adaptive selection of the task branch, activating only the recognition tasks that are most valuable for the current image, improving the computational efficiency and model response speed. In this way, the system can automatically and intelligently determine the computing path and task activation state according to the image content, avoiding the waste of resources and performance bottlenecks caused by "one-size-fits-all", and realizing the technological innovation in the field of image recognition.

[0041] Specifically, the feature recognition model corresponding to the food image to be identified is determined according to the difficulty score scalar, including: constructing a feature recognition model sequence, wherein the feature recognition model sequence is composed of a plurality of feature recognition models arranged from low to high according to the model network layer number; constructing a model matching table, wherein the model matching table is a mapping relationship table between the model network layer number of the feature recognition model and the difficulty score scalar; selecting the feature recognition model corresponding to the food image to be identified from the feature recognition model sequence according to the difficulty score scalar and the model matching table.

[0042] The process of determining the feature recognition model corresponding to the food image to be identified according to the difficulty score scalar includes constructing a feature recognition model sequence composed of a plurality of feature recognition models arranged from low to high according to the model network layer number; and constructing a model matching table to provide a basis for the mapping relationship between the model network layer number of the feature recognition model and the difficulty score scalar. This process can intelligently select the most suitable model for processing the current image from the feature recognition model sequence by combining the use of the difficulty score scalar and the model matching table. This design mechanism ensures efficient use of computing resources, complex images can be given sufficient network depth to ensure recognition accuracy, and simple images will be allocated to shallower models to avoid unnecessary computation, thereby significantly improving processing speed and system responsiveness while ensuring recognition accuracy.

[0043] More specifically, determining the recognition result of the to-be-recognized food image according to the task relevance vector and the recognition feature vector includes: determining the task recognition type of the to-be-recognized food image according to the task relevance vector; selecting a task calculation model corresponding to the to-be-recognized food image based on the task recognition type, and inputting the recognition feature vector into the task calculation model to obtain the recognition result corresponding to the to-be-recognized food image.

[0044] The task recognition type includes food classification, food material segmentation, calorie estimation, etc.

[0045] Further, determining the task recognition type of the to-be-recognized food image according to the task relevance vector includes: inputting the task relevance vector into a Gumbel-Softmax-based sampler to obtain a task activation mask, and determining the task recognition type corresponding to the to-be-recognized food image according to the task activation mask.

[0046] The adaptive task activation process of the method uses the Gumbel-Softmax trick to sample the task relevance vector to obtain a task activation mask, and determines the activated task branch according to the mask value, thereby realizing personalized processing of the input food image and improving the pertinence and effectiveness of task execution. In this way, the system can accurately identify the food category, food material distribution, and calorie information in the image, while significantly reducing unnecessary computational overhead and improving the utilization efficiency of computing resources.

[0047] Further, the multi-layer perceptron in the recognition difficulty perception model includes a hidden layer and an output layer of a preset dimension, the hidden layer adopts a ReLU activation function, the output layer includes a first output layer and a second output layer, the first output layer is used to output a difficulty score scalar, the first output layer adopts a Sigmoid activation function, the second output layer is used to output a task relevance vector, and the second output layer adopts a Softmax activation function.

[0048] The method multi-layer perception (MLP) is configured to have a specific hidden layer and an output layer (Difficulty Perception Module, DPM). The hidden layer adopts a ReLU activation function, effectively enhancing the non-linear expression ability of the model, enabling it to capture complex patterns in the image more subtly. The first output layer is responsible for generating a difficulty score scalar and using a Sigmoid activation function to limit its output range to between 0 and 1, intuitively reflecting the difficulty of image processing. The second output layer outputs a task relevance vector through a Softmax activation function, ensuring that the vector elements sum to 1, facilitating the selection of the subsequent adaptive task activation mechanism. This design not only improves the accuracy of task selection, but also ensures that the system can automatically adjust the calculation process according to the specific content of the image, achieving efficient use of computing resources. The implementation effect shows that this method can dynamically adjust the computational complexity of the food recognition system without sacrificing recognition accuracy, significantly improving efficiency and adaptability in various application scenarios.

[0049] Specifically, the feature extraction processing is performed on the to-be-identified food image to obtain a shared feature map, including: constructing a lightweight feature extraction model, inputting the to-be-identified food image into the lightweight feature extraction model to obtain the shared feature map, wherein the lightweight feature extraction model is a lightweight convolutional neural network model.

[0050] The lightweight feature extraction model is a lightweight shared feature extraction backbone network, such as the first few layers of a lightweight CNN or MobileNet, responsible for receiving and initially analyzing image data to generate a shared feature map as the basis for subsequent processing.

[0051] In order for those skilled in the art to more clearly understand the technical solutions of the present application, the implementation process of the food image recognition method of the present application will be described in detail below in conjunction with specific embodiments.

[0052] The present embodiment relates to a specific food image recognition method, as shown in Figure 3 The shallow feature extraction module extracts the shallow features of the input image, and according to the shallow features, the difficulty perception and task selection module determines the task relevance vector and the difficulty score scalar. According to the difficulty score scalar, the corresponding deep feature extraction module is selected to extract the high-level features of the input image, and the shared feature extraction module is used to share the high-level features to obtain the shared features. According to the shared features and the task relevance vector, the activated task is executed to obtain the recognition result of the input image. Specifically, the following steps are included:

[0053] Step one: image input and primary feature extraction: input the input food image I into a lightweight shared feature extraction backbone network (e.g. the first few layers of a lightweight CNN or MobileNet) to get initial shared feature maps : ;

[0054] Step 2: Image processing difficulty perception and prediction:

[0055] The shared feature maps are input into a difficulty perception module (DPM). The module consists of a global average pooling layer (GAP) and a multi-layer perceptron (MLP), and its output is a difficulty score scalar and a task relevance vector , where is the total number of tasks supported by the system (such as classification, segmentation, detection, etc.). The closer the difficulty score is to 1, the more complex the image is, and the more computational resources are needed. The closer it is to 0, the simpler the image is. Each element of the task relevance vector represents the relevance probability of the image to the th task.

[0056] Step 3: Difficulty-based dynamic depth selection;

[0057] The system predefines a set of subnetworks with different depths , where is the shallowest and simplest, is the deepest and most complex. A resource allocation controller (RAC) is used to dynamically select the most suitable subnetwork based on the difficulty score for subsequent deep feature extraction.

[0058] , where , for example, if (4 subnetworks), then select ; if , then select .

[0059] Subsequently, the shared features are input into the selected subnetwork to obtain high-level features .

[0060] .

[0061] Step four: Adaptive task activation based on relevance;

[0062] high-level features are input to a shared head , which further extracts task-shared features . .

[0063] The system predefines a dedicated task branch T_k for each task k (e.g., a fully connected layer for classification and a decoder for segmentation), not all branches will be activated. The task relevance vector goes through a Gumbel-Softmax-based sampler to generate a sparse task activation mask , where only the top tasks (configurable) with the highest probability have a mask value of 1, and the rest are 0.

[0064] Finally, only the activated task branches will be calculated to produce the output . .

[0065] Step five: output and integration:

[0066] Collect the outputs of all activated task branches as the final recognition result of the system.

[0067] The difficulty perception module (DPM), resource allocation controller (RAC), shared backbone, subnetwork, and task branch are jointly trained in an end-to-end manner. The training strategy includes two loss functions:

[0068] Main Loss: The weighted sum of the standard losses (such as cross-entropy) of all tasks, but only the loss of the activated branch is calculated.

[0069] Collaborative Loss: To guide the difficulty prediction, an auxiliary loss is introduced. This loss encourages the difficulty score d to be positively correlated with the actual computational cost (such as FLOPs) calculated on the selected subnetwork of the image, and positively correlated with the model's prediction uncertainty (such as classification entropy) on the image. At the same time, it encourages the task relevance vector to be consistent with the output confidence of each task branch.

[0070] where, ​The first 5 reverse residual blocks of MobileNetV3 can be optionally used.

[0071] The MLP in can be designed as two layers, with a 128-dimensional hidden layer using ReLU activation, and the output layer using Sigmoid (for ) and Softmax (for ) activation, respectively.

[0072] The subnetwork set may include 4 options: (2 reverse residual blocks), (4 blocks), (8 blocks), and B4 (16 blocks).

[0073] The task can be set as: (food classification), (food material segmentation), (calorie estimation).

[0074] During training, the AdamW optimizer is used with an initial learning rate of 3e-4, and the warm-up and cosine decay strategies are adopted. The weight coefficient of the collaborative loss is set to 0.1.

[0075] The embodiment specifically realizes the following effects:

[0076] 1. Significantly improve efficiency: by bypassing unnecessary deep calculations and irrelevant tasks, the average inference speed is improved by 40%-60%, and the computing resource consumption is reduced by 35%-55%, which is particularly beneficial to resource-constrained environments.

[0077] 2. Maintain high accuracy: complex images still get sufficient computing resources, ensuring that the overall recognition accuracy is not compromised, and even slightly improved on simple images due to reduced overfitting risk.

[0078] 3. Intelligent and adaptive: the system does not require human intervention and can automatically make decisions based on image content, realizing an intelligent computing paradigm of on-demand allocation.

[0079] The embodiment of the present application also provides a food image recognition device. It should be noted that the food image recognition device of the embodiment of the present application can be used to execute the food image recognition method provided by the embodiment of the present application. The device is used to realize the above-mentioned embodiments and preferred embodiments, and will not be described again. As used below, the term "module" can be a combination of software and / or hardware that implements a predetermined function. Although the device described in the following embodiments is preferably implemented in software, hardware, or a combination of software and hardware can also be implemented and conceived.

[0080] The food image recognition device provided by the embodiment of the present application is introduced as follows.

[0081] Figure 4 is a schematic diagram of the food image recognition device according to the embodiment of the present application. As shown in the figure, the device comprises: Figure 4

[0082] The acquisition unit 41 is configured to acquire a food image to be recognized, perform feature extraction processing on the food image to be recognized, and obtain a shared feature map.

[0083] The first determination unit 42 is configured to determine the difficulty of recognizing the food image to be recognized and the task recognition type based on the shared feature map, obtain a difficulty score scalar and a task relevance vector, and determine a feature recognition model corresponding to the food image to be recognized according to the difficulty score scalar.

[0084] The second determination unit 43 is configured to input the shared feature map into the feature recognition model corresponding to the food image to be recognized, obtain a recognition feature vector, and determine the recognition result of the food image to be recognized according to the task relevance vector and the recognition feature vector.

[0085] ​In this embodiment, the acquisition unit is used to acquire an image of the food to be identified, perform feature extraction processing on the image to obtain a shared feature map; the first determination unit is used to determine the recognition difficulty and task recognition type of the food image to be identified based on the shared feature map, obtain a difficulty score scalar and a task relevance vector, and determine the feature recognition model corresponding to the food image to be identified based on the difficulty score scalar; the second determination unit is used to input the shared feature map into the feature recognition model corresponding to the food image to be identified, obtain a recognition feature vector, and determine the recognition result of the food image to be identified based on the task relevance vector and the recognition feature vector. By extracting the primary features (shared feature map) of the input food image, the processing difficulty of the image is then evaluated, and the task type most suitable for the current image recognition needs is identified. This evaluation result determines the computational resource level required for subsequent deep feature extraction and the activation state of the task branch. Based on the image's difficulty score scalar, the system intelligently selects the most suitable deep computation model, avoiding resource waste caused by over-computation of simple images. Furthermore, based on the task relevance vector, the system can accurately activate task branches that highly match the image task, ensuring that complex images receive sufficient computational resources while eliminating interference from irrelevant tasks, thus improving overall recognition accuracy and efficiency. In summary, the dynamic computation graph and difficulty-adaptive recognition scheme proposed in this embodiment achieves a reasonable allocation of computational resources, improving the overall performance of food image recognition. While maintaining recognition accuracy, it significantly reduces computational resource consumption, thereby solving the problem of low recognition efficiency in existing food image recognition schemes.

[0086] As an optional scheme, the first determining unit includes a first construction module and a first input module; the first construction module is used to construct a recognition difficulty perception model, wherein the recognition difficulty perception model is composed of a global average pooling layer and a multi-layer perception mechanism; the first input module is used to input the shared feature map into the recognition difficulty perception model to obtain the difficulty score scalar and the task correlation vector.

[0087] In one optional scheme, the first determining unit further includes a second construction module, a third construction module, and a selection module; the second construction module is used to construct a feature recognition model sequence, wherein the feature recognition model sequence consists of multiple feature recognition models arranged in ascending order of model network layer number; the third construction module is used to construct a model matching table, wherein the model matching table is a mapping relationship table between the model network layer number of the feature recognition model and the difficulty score scalar; the selection module is used to select the feature recognition model corresponding to the food image to be recognized from the feature recognition model sequence according to the difficulty score scalar and the model matching table.

[0088] In an optional implementation, the second determining unit includes a first determining module and a second input module; the first determining module is configured to determine the task recognition type of the to-be-identified food image according to the task correlation vector; and the second input module is configured to select a task calculation model corresponding to the to-be-identified food image based on the task recognition type, and input the recognition feature vector into the task calculation model to obtain the recognition result corresponding to the to-be-identified food image.

[0089] In an optional implementation, the first determining module includes a determining submodule configured to input the task correlation vector into a Gumbel-Softmax-based sampler to obtain a task activation mask, and determine the task recognition type corresponding to the to-be-identified food image according to the task activation mask.

[0090] In an optional implementation, the multi-layer perceptron in the recognition difficulty perception model includes a preset-dimension hidden layer and an output layer; the hidden layer adopts a ReLU activation function; the output layer includes a first output layer and a second output layer; the first output layer is configured to output a difficulty score scalar, and adopts a Sigmoid activation function; and the second output layer is configured to output a task correlation vector, and adopts a Softmax activation function.

[0091] In an optional implementation, the obtaining unit includes a fourth constructing module configured to construct a lightweight feature extraction model, and input the to-be-identified food image into the lightweight feature extraction model to obtain the shared feature map; and the lightweight feature extraction model is a lightweight convolutional neural network model.

[0092] The food image recognition device includes a processor and a memory; the obtaining unit, the first determining unit, the second determining unit, and the like are stored in the memory as program units, and the processor executes the program units stored in the memory to implement corresponding functions. The modules are located in the same processor; or, the modules are located in different processors in any combination.

[0093] The processor includes a core, and the core retrieves corresponding program units from the memory. The core can be one or more, and the core parameters can be adjusted to solve the problem of low recognition efficiency in existing food image recognition schemes.

[0094] The memory can include a non-permanent memory in a computer readable medium, a random access memory (RAM), and / or a non-volatile memory such as a read-only memory (ROM) or a flash memory (flash RAM), and the memory includes at least one memory chip.

[0095] The embodiment of the present application provides a computer readable storage medium, the computer readable storage medium comprises a stored program, wherein the program controls the device where the computer readable storage medium is located to execute the food image recognition method when the program is running.

[0096] Specifically, the food image recognition method comprises:

[0097] In step S201, a food image to be recognized is acquired, feature extraction processing is performed on the food image to be recognized, and a shared feature map is obtained.

[0098] In step S202, the difficulty score scalar and the task relevance vector are obtained by determining the difficulty of recognizing the food image to be recognized and the task recognition type based on the shared feature map, and the feature recognition model corresponding to the food image to be recognized is determined according to the difficulty score scalar, wherein the difficulty score scalar is related to the computing resources required for recognizing the food image to be recognized.

[0099] In step S203, the shared feature map is input into the feature recognition model corresponding to the food image to be recognized, an identification feature vector is obtained, and the recognition result of the food image to be recognized is determined according to the task relevance vector and the identification feature vector.

[0100] The embodiment of the present application provides a processor, the processor is used for running a program, wherein the processor executes the food image recognition method when the program is running.

[0101] Specifically, the food image recognition method comprises:

[0102] In step S201, a food image to be recognized is acquired, feature extraction processing is performed on the food image to be recognized, and a shared feature map is obtained.

[0103] In step S202, the difficulty score scalar and the task relevance vector are obtained by determining the difficulty of recognizing the food image to be recognized and the task recognition type based on the shared feature map, and the feature recognition model corresponding to the food image to be recognized is determined according to the difficulty score scalar, wherein the difficulty score scalar is related to the computing resources required for recognizing the food image to be recognized.

[0104] In step S203, the shared feature map is input into the feature recognition model corresponding to the food image to be recognized, an identification feature vector is obtained, and the recognition result of the food image to be recognized is determined according to the task relevance vector and the identification feature vector.

[0105] The embodiment of the present application provides an electronic device, the device comprises a processor, a memory and a program stored in the memory and capable of running on the processor, and the processor executes the program to realize at least the following steps:

[0106] In step S201, an image of a food to be identified is acquired, and a shared feature map is obtained by performing feature extraction on the image of the food to be identified.

[0107] In step S202, a difficulty score scalar and a task relevance vector are obtained by determining a difficulty of identifying the image of the food to be identified and a task type based on the shared feature map, and a feature recognition model corresponding to the image of the food to be identified is determined according to the difficulty score scalar, where the difficulty score scalar is related to a computing resource required for identifying the image of the food to be identified.

[0108] In step S203, an identification result of the image of the food to be identified is determined according to the task relevance vector and an identification feature vector obtained by inputting the shared feature map into the feature recognition model corresponding to the image of the food to be identified.

[0109] The device herein can be a server, a PC, a PAD, a mobile phone, or the like.

[0110] The present application also provides a computer program product adapted to execute a program including at least the following method steps when executed on a data processing device:

[0111] In step S201, an image of a food to be identified is acquired, and a shared feature map is obtained by performing feature extraction on the image of the food to be identified.

[0112] In step S202, a difficulty score scalar and a task relevance vector are obtained by determining a difficulty of identifying the image of the food to be identified and a task type based on the shared feature map, and a feature recognition model corresponding to the image of the food to be identified is determined according to the difficulty score scalar, where the difficulty score scalar is related to a computing resource required for identifying the image of the food to be identified.

[0113] In step S203, an identification result of the image of the food to be identified is determined according to the task relevance vector and an identification feature vector obtained by inputting the shared feature map into the feature recognition model corresponding to the image of the food to be identified.

[0114] It should be apparent to those skilled in the art that the modules or steps of the application described above can be implemented with a general purpose computer, and can be centralized in a single computer or distributed among a network of computers, and can be implemented with program code executable by a computer, and thus can be stored in a storage device and executed by a computer, and in some cases, the steps shown or described can be executed in a different order than shown or described, or can be implemented as separate integrated circuit modules or as a single integrated circuit module, and thus the application is not limited to any particular combination of hardware and software.

[0115] Those skilled in the art will appreciate that embodiments of the application can be devised for a method, a system, or a computer program product. Accordingly, the application can be embodied in the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the application can take the form of a computer program product on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROMs, optical storage devices, etc.) embodying computer readable program code.

[0116] The present application is described herein with reference to the flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart illustrations and / or block diagrams. Figure 1 one or more functions specified in the flowchart illustrations and / or block diagrams. Figure 1 one or more functions specified in the flowchart illustrations and / or block diagrams.

[0117] These computer program instructions can also be stored in a computer- readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instructions which implement the functions specified in the flowchart illustrations and / or block diagrams. Figure 1 one or more functions specified in the flowchart illustrations and / or block diagrams. Figure 1 one or more functions specified in the flowchart illustrations and / or block diagrams.

[0118] These computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart illustrations and / or block diagrams.Figure 1 one or more processes and / or blocks Figure 1 the function(s) specified in the flow block or blocks.

[0119] In one typical arrangement, the computing device includes one or more processors (CPUs), input / output interfaces, network interfaces, and memory.

[0120] The memory can include non-persistent memory and / or volatile memory, such as random access memory (RAM) about which the processor can read instructions stored thereon for execution; and / or non-volatile memory, such as read only memory (ROM), electrically programmable read only memory (EPROM), electrically erasable programmable read only memory (EEPROM), flash memory, or other solid state memory technology. The memory is an example of computer readable media.

[0121] Computer readable media includes permanent and non-permanent, removable and non-removable media implemented in any method or technology for storage of information such as computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read only memory (ROM), electrically programmable read only memory (EEPROM), flash memory or other memory technology, compact disc read only memory (CD-ROM), digital versatile disc (DVD), or other optical storage, magnetic cassettes, magnetic tapes, magnetic disk storage or other magnetic storage devices, or any other non-transmission medium that can be used to store information accessible to a computing device. According to the definition herein, computer readable media does not include transitory media such as modulated data signals and carrier waves.

[0122] The technical features of the above-described embodiments can be combined in any manner, and in order to make the description concise, not all possible combinations of the technical features in the above-described embodiments are described, however, as long as the combinations of the technical features do not contradict each other, it should be considered that they are within the scope of the present disclosure.

[0123] It should also be noted that the terms "comprising", "containing", or any other variant thereof are intended to encompass non-exclusive inclusion, such that processes, methods, articles or apparatuses that include a series of elements not only include those elements, but also include other elements not explicitly listed or inherent to such processes, methods, articles or apparatuses. Without more limitations, the element defined by the statement "comprising a" does not exclude the presence of additional identical elements in the process, method, article or apparatus that includes the element.

[0124] The above descriptions are only the preferred embodiments of the present application, and are not intended to limit the present application. The present application can have various modifications and changes for those skilled in the art. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A method for recognizing food images, characterized in that, include: Acquire an image of the food to be identified, and perform feature extraction processing on the image to obtain a shared feature map; Based on the shared feature map, the recognition difficulty and task recognition type of the food image to be recognized are determined, a difficulty score scalar and a task relevance vector are obtained, and a feature recognition model corresponding to the food image to be recognized is determined according to the difficulty score scalar, wherein the difficulty score scalar is related to the computing resources required to recognize the food image to be recognized; The shared feature map is input into the feature recognition model corresponding to the food image to be recognized to obtain the recognition feature vector, and the recognition result of the food image to be recognized is determined according to the task relevance vector and the recognition feature vector.

2. The method according to claim 1, characterized in that, Based on the shared feature map, the recognition difficulty and task recognition type of the food image to be recognized are determined, resulting in a difficulty score scalar and a task relevance vector, including: A recognition difficulty perception model is constructed, wherein the recognition difficulty perception model is composed of a global average pooling layer and a multi-layer perception mechanism; The shared feature map is input into the recognition difficulty perception model to obtain the difficulty score scalar and the task relevance vector.

3. The method according to claim 1, characterized in that, Determining the feature recognition model corresponding to the food image to be identified based on the difficulty score scalar includes: Construct a feature recognition model sequence, wherein the feature recognition model sequence consists of multiple feature recognition models arranged in ascending order of model network layer number; Construct a model matching table, wherein the model matching table is a mapping relationship table between the number of network layers of the feature recognition model and the difficulty score scalar; Based on the difficulty score scalar and the model matching table, the feature recognition model corresponding to the food image to be recognized is selected from the feature recognition model sequence.

4. The method according to claim 1, characterized in that, Determining the recognition result of the food image to be recognized based on the task relevance vector and the recognition feature vector includes: The task recognition type of the food image to be recognized is determined based on the task relevance vector; Based on the task recognition type, a task calculation model corresponding to the food image to be recognized is selected, and the recognition feature vector is input into the task calculation model to obtain the recognition result corresponding to the food image to be recognized.

5. The method according to claim 4, characterized in that, Determining the task recognition type of the food image to be recognized based on the task relevance vector includes: The task relevance vector is input into a Gumbel-Softmax-based sampler to obtain a task activation mask, and the task recognition type corresponding to the food image to be recognized is determined based on the task activation mask.

6. The method according to claim 2, characterized in that, The multilayer perceptron in the difficulty perception model includes a hidden layer and an output layer of preset dimensions. The hidden layer uses the ReLU activation function, and the output layer includes a first output layer and a second output layer. The first output layer is used to output a difficulty score scalar and uses the Sigmoid activation function. The second output layer is used to output a task relevance vector and uses the Softmax activation function.

7. The method according to claim 1, characterized in that, The image of the food to be identified is subjected to feature extraction processing to obtain a shared feature map, including: A lightweight feature extraction model is constructed. The food image to be identified is input into the lightweight feature extraction model to obtain the shared feature map. The lightweight feature extraction model is a lightweight convolutional neural network model.

8. A food image recognition device, characterized in that, include: An acquisition unit is used to acquire an image of the food to be identified, and to perform feature extraction processing on the image of the food to be identified to obtain a shared feature map; The first determining unit is used to determine the recognition difficulty and task recognition type of the food image to be recognized based on the shared feature map, obtain a difficulty score scalar and a task relevance vector, and determine the feature recognition model corresponding to the food image to be recognized based on the difficulty score scalar. The second determining unit is used to input the shared feature map into the feature recognition model corresponding to the food image to be recognized, obtain the recognition feature vector, and determine the recognition result of the food image to be recognized based on the task relevance vector and the recognition feature vector.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes a stored program, wherein, when the program is executed, it controls the device on which the computer-readable storage medium is located to perform the food image recognition method according to any one of claims 1 to 7.

10. An electronic device, characterized in that, include: One or more processors, a memory, and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, the one or more programs including a method for performing the food image recognition method according to any one of claims 1 to 7.

Images