Food testing methods and related equipment

Abstract

The present invention provides a food detection method and related equipment, the method comprising: inputting a first food image to be detected into a recognition model, wherein the training of the recognition model includes the training of a first gradient value of a food recognition task and the training of a second gradient value of a food freshness recognition task, and the first gradient value and the second gradient value satisfy a gradient normalization condition during the training process; and determining the food in the first food image and the freshness of the food according to the output result of the recognition model. In the present invention, the training of the recognition model includes the training of a first gradient value of a food recognition task and the training of a second gradient value of a food freshness recognition task, and the first gradient value and the second gradient value satisfy a gradient normalization condition during the training process, preventing the model from focusing on training tasks with larger gradients, thereby avoiding focusing on training on food categories or food freshness, and improving the model's recognition accuracy for food categories and freshness.

Description

Technical Field

[0001] The present invention relates to the technical field of image recognition, and in particular to a food detection method and related equipment. Background Art

[0002] As people pay more attention to their physical health, fruits and vegetables have become one of the essential foods in people's daily life.

[0003] When buying fruits and vegetables, people may buy enough for a few days, but fruits and vegetables have a short shelf life and people cannot judge their freshness.

[0004] In the exemplary technology, the freshness of vegetables and fruits can be identified by deep learning, that is, the freshness of vegetables and fruits can be identified by the model. However, the model needs to identify the category and freshness of food at the same time. During the training process of the model, the training of the food category or food freshness will be emphasized, which will cause errors in the model's identification of the food category or food freshness, that is, the identification of food freshness and category has the problem of low recognition accuracy. Summary of the invention

[0005] The present invention provides a food detection method and related equipment, which are used to solve the problem of low recognition accuracy in the recognition of food freshness and category.

[0006] In one aspect, the present invention provides a food detection method, comprising:

[0007] Inputting a first food image to be detected into a recognition model, wherein the training of the recognition model includes the training of a first gradient value of a food recognition task and the training of a second gradient value of a food freshness recognition task, and the first gradient value and the second gradient value satisfy a gradient normalization condition during the training process;

[0008] The food in the first food image and the freshness of the food are determined according to the output result of the recognition model.

[0009] In one embodiment, before the step of inputting the food image to be detected into the recognition model, the method further includes:

[0010] Acquire each training image, wherein the training image includes a food label and a food freshness label;

[0011] According to the food labels and food freshness labels of each of the training images, multiple rounds of training are performed on the first gradient value and the second gradient value of the multi-task model to obtain a recognition model, wherein the first gradient value, the first training weight corresponding to the first gradient value, the second gradient value, and the second training weight corresponding to the second gradient value in each round meet the gradient normalization condition.

[0012] In one embodiment, the step of acquiring each training image includes:

[0013] acquiring each second food image;

[0014] Performing data enhancement processing on each of the second food images to obtain each of the third food images, wherein the enhancement processing includes contrast transformation, grayscale change and / or scale transformation;

[0015] A food label and a food freshness label are added to each of the third food images to obtain each of the training images.

[0016] In one embodiment, the step of performing multiple rounds of training on the first gradient value and the second gradient value of the multi-task model according to the food label and the food freshness label of each training image to obtain the recognition model includes:

[0017] Setting initial training parameters for the multi-task model, wherein the initial training parameters are model parameters of a preset model that has completed training;

[0018] According to the food labels and food freshness labels of each of the training images, multiple rounds of training are performed on the first gradient value and the second gradient value of the multi-task model for setting initial training parameters.

[0019] In one embodiment, the first training weight is determined according to a learning rate of the food recognition task, and the second training weight is determined according to a learning rate of the food freshness recognition task.

[0020] In one embodiment, the multi-task model includes a first decoder and a second decoder arranged in parallel, the first decoder and the second decoder each include three fully connected layers and one output layer, the first decoder is used to identify the food in the food image, and the second decoder is used to identify the freshness of the food in the food image.

[0021] In another aspect, the present invention further provides a food detection device, comprising:

[0022] An input module, used for inputting a first food image to be detected into a recognition model, wherein the training of the recognition model includes the training of a first gradient value of a food recognition task and the training of a second gradient value of a food freshness recognition task, and the first gradient value and the second gradient value satisfy a gradient normalization condition during the training process;

[0023] A determination module is used to determine the food in the first food image and the freshness of the food according to the output result of the recognition model.

[0024] In another aspect, the present invention further provides a food detection device, comprising: a memory and a processor;

[0025] The memory stores computer-executable instructions;

[0026] The processor executes the computer-executable instructions stored in the memory, so that the processor performs the food detection method as described above.

[0027] In one embodiment, the food detection device is a refrigerator.

[0028] On the other hand, the present invention further provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer-executable instructions, and when the computer-executable instructions are executed by a processor, they are used to implement the food detection method as described above.

[0029] The food detection method and related equipment provided by the present invention input the food image to be detected into the recognition model, so as to determine the food in the food image and the freshness of the food according to the output result of the recognition model. In the present invention, the training of the recognition model includes the training of the first gradient value of the food recognition task and the training of the second gradient value of the food freshness recognition task, and the first gradient value and the second gradient value meet the gradient normalization condition during the training process, so as to prevent the model from focusing on the training of tasks with larger gradients, thereby avoiding focusing on the training of food categories or food freshness, and improving the recognition accuracy of the model for food categories and freshness. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present disclosure and, together with the description, serve to explain the principles of the present disclosure.

[0031] Figure 1 A schematic diagram of a food detection method according to the present invention;

[0032] Figure 2 It is a schematic diagram of the process of the first embodiment of the food detection method of the present invention;

[0033] Figure 3 is a schematic flow chart of a second embodiment of a food detection method of the present invention;

[0034] Figure 4 This is a detailed flow chart of step S30 in the third embodiment of the food detection method of the present invention;

[0035] Figure 5 is a schematic diagram of a module of a food detection device of the present invention;

[0036] Figure 6 It is a structural schematic diagram of the food detection device of the present invention.

[0037] The above drawings show clear embodiments of the present disclosure, which will be described in more detail below. These drawings and text descriptions are not intended to limit the scope of the present disclosure in any way, but to illustrate the concepts of the present disclosure to those skilled in the art by referring to specific embodiments. DETAILED DESCRIPTION

[0038] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present disclosure. Instead, they are merely examples of devices and methods consistent with some aspects of the present disclosure as detailed in the appended claims.

[0039] The present invention provides a food detection method. Figure 1 As shown, the food detection device 100 can be a device for storing fruits and vegetables, such as a refrigerator, or a device with image recognition. The food detection device 100 can capture the image 110, or the image 110 can be input into the food detection device 100. A recognition model (not shown) is also provided in the food detection device 100. The food detection device 100 inputs the image 110 into the recognition model, and the recognition model outputs the food in the image 110 and the freshness of the food. The food detection device 100 can output the food and the freshness of the food. For example, if the food detection device recognizes that the food in the image 110 is green vegetables, and the freshness is level one, it will output "green vegetables, level one freshness."

[0040] The following specific embodiments are used to describe in detail the technical solutions of the present invention and how the technical solutions of the present application solve the above-mentioned technical problems. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of the present invention will be described below in conjunction with the accompanying drawings.

[0041] Reference Figure 2 , Figure 2 The first embodiment of the food detection method of the present invention comprises the following steps:

[0042] Step S10, inputting the first food image to be detected into the recognition model, wherein the training of the recognition model includes the training of the first gradient value of the food recognition task and the training of the second gradient value of the food freshness recognition task, and the first gradient value and the second gradient value satisfy the gradient normalization condition during the training process.

[0043] In this embodiment, the execution subject is a food detection device, which can be a device capable of storing food, such as a refrigerator, or a device capable of image recognition. For ease of description, the term "device" is used below to refer to the food detection device.

[0044] The device obtains a food image to be detected, which is defined as a first food image. The device is provided with a recognition model, which is used to recognize food in the food image and the freshness of the food.

[0045] In this embodiment, the recognition model is a model obtained through multi-task learning. The multi-task includes a food recognition task and a food freshness recognition task. Based on multi-task learning, the training of the recognition model includes the training of the first gradient value of the food recognition task and the training of the second gradient value of the food freshness recognition task, and the first gradient value and the second gradient value meet the gradient normalization condition during the training process. The gradient normalization condition is, for example: setting corresponding coefficients for the first gradient value and the second gradient value, the coefficients determine the degree of training for the first gradient value and the second gradient value, and adaptively modifying the corresponding coefficients of the first gradient value and the second gradient value based on the training results during the training process, so that the first gradient value, the second gradient value, and the corresponding coefficients of the two are fixed values.

[0046] Step S20: determining the food in the first food image and the freshness of the food according to the output result of the recognition model.

[0047] After the first food image is input into the recognition model, the recognition model outputs a result, and the device can determine the food in the first food image and the freshness of the food according to the output result. The freshness can be, for example, fresh, first-grade sub-fresh, second-grade sub-fresh, first-grade rotten, second-grade rotten, etc.

[0048] The device may have an alarm condition, and if the alarm condition is met, the device may output an alarm message. For example, if the alarm condition is that the freshness of the food is level one or level two of rottenness, the alarm message is output.

[0049] In the technical solution provided in this embodiment, the food image to be detected is input into the recognition model, so as to determine the food in the food image and the freshness of the food according to the output result of the recognition model. In the present invention, the training of the recognition model includes the training of the first gradient value of the food recognition task and the training of the second gradient value of the food freshness recognition task, and the first gradient value and the second gradient value meet the gradient normalization condition during the training process, so as to prevent the model from focusing on the training of tasks with larger gradients, thereby avoiding focusing on the training of food categories or food freshness, and improving the recognition accuracy of the model for food categories and freshness.

[0050] Reference Figure 3 , Figure 3 This is a second embodiment of the food detection method, based on the first embodiment, and before step S10, further comprising:

[0051] Step S30, obtaining each training image, wherein the training image includes a food label and a food freshness label.

[0052] In this embodiment, the device obtains each training image, which is manually labeled and is labeled with food labels and food freshness labels.

[0053] Step S40, according to the food labels and food freshness labels of each training image, perform multiple rounds of training on the first gradient value and the second gradient value of the multi-task model to obtain a recognition model, wherein the first gradient value of each round, the first training weight corresponding to the first gradient value, the second gradient value, and the second training weight corresponding to the second gradient value meet the gradient normalization condition.

[0054] In this embodiment, a multi-task learning framework is used for model training. The multi-task learning framework can be constructed using a supervised learning model. The model trained in the multi-task learning framework is defined as a multi-task model. The device performs multiple rounds of training on the first gradient value and the second gradient value of the multi-task model based on the food labels and food freshness labels of each training image. When the first gradient value and the second gradient value reach the corresponding target value of each image, the multi-task model training is completed. In addition, the multi-task model needs to learn two different tasks, and the multi-task model is provided with two loss functions for food recognition and freshness detection. After the first gradient value and the second gradient value are trained, the difference between the first gradient value and the normalized gradient value can be set as the loss function of food recognition, and the difference between the second gradient value and the normalized gradient value can be set as the loss function of food freshness. The loss function for food recognition can be categorical cross entropy, and the loss function for food freshness recognition can be binary cross entropy.

[0055] In this embodiment, the gradient normalization condition means that the operation of the first gradient value, the first training weight corresponding to the first gradient value, the second gradient value, and the second training weight corresponding to the second gradient value is a fixed value, and the fixed value is the normalized gradient value.

[0056] For example,

[0057]

[0058] Among them, L is the normalized gradient value, ω i is the training weight of the i-th gradient value, L i is the i-th gradient value,

[0059] It can be understood that the product of the first gradient value and the first training weight plus the product of the second gradient value and the second training weight needs to be equal to the normalized gradient value.

[0060] It should be noted that the first gradient value and the second gradient value need to undergo multiple rounds of training, and in each round of training, the first training weight, the second gradient value, and the second training weight corresponding to the second gradient value meet the gradient normalization condition. In each round of training, the first gradient value and the second gradient value will be dynamically adjusted, and the second training weight and the first training weight will also be adaptively modified. The first training weight and the second training weight are adaptively modified according to the training results of each corresponding task, that is, the first training weight and the second training weight are adaptively modified according to the learning rate of the task, so that the training gradient value of each task moves in the direction of gradient normalization. In this embodiment, the first training weight is determined according to the learning rate of the food recognition task, and the second training weight is determined according to the learning rate of the food freshness task.

[0061] In the technical solution provided in this embodiment, the device obtains each training image, and then performs multiple rounds of training on the first gradient value and the second gradient value of the multi-task model according to the food labels and food freshness labels of the training images to obtain a recognition model. Since the first gradient value, the first training weight corresponding to the first gradient value, the second gradient value, and the second training weight corresponding to the second gradient value in each round of training meet the gradient normalization condition, the recognition model can accurately identify the category of food and the freshness of food.

[0062] In one embodiment, the multi-task includes an encoder and a decoder. The encoder is mainly composed of a convolution layer and a pooling layer. The convolution layer is mainly used to extract feature information of food images; while the pooling layer can speed up the calculation speed and prevent overfitting.

[0063] In this embodiment, the encoder also includes an encoding part, which is composed of a residual convolutional network module, which fuses the output of each layer with the output of the previous layer through a layer-selection connection operation to prevent feature dissipation during the training process. The encoder can extract data from the two-dimensional image representation information and improve the feature extraction capability of the model, which can solve the problem of inaccurate recognition caused by factors such as food being blocked in the food image.

[0064] In addition, the decoder of the multi-task model includes a first decoder and a second decoder. The first decoder and the second decoder are arranged in parallel, that is, the information output by the encoder is input into the first decoder and the second decoder respectively, the first decoder is used to identify food in food images, and the second decoder is used to identify the freshness of food in food images. The first decoder and the second decoder both include three layers of full connection and one output layer, and the output layer can be a Softmax layer. The fully connected layer mainly plays a role of classification, and the fully connected layer is used to obtain the probability of the food to which the input data belongs.

[0065] In addition, in this embodiment, the parameters of the multi-task recognition model need to be iteratively optimized, and the optimizer for the optimization may be an Adam optimizer.

[0066] In this embodiment, the multi-task recognition model includes a first encoder and a second encoder that are arranged in parallel, so as to identify the category of food and the freshness of food in the food image.

[0067] Reference Figure 4 , Figure 4 This is the third embodiment of the food detection method of the present invention. Based on the second embodiment, step S30 includes:

[0068] Step S31, acquiring each second food image.

[0069] Step S32, performing data enhancement processing on each second food image to obtain each third food image, wherein the enhancement processing includes contrast transformation, grayscale change and / or scale transformation.

[0070] In this embodiment, in order to increase the recognition and detection robustness of the model, it is necessary to perform data expansion on the original data.

[0071] In this regard, the device obtains multiple food images, which are defined as second food images. The device then performs enhancement processing on the second food image to obtain a third food image, and the enhancement processing includes but is not limited to: contrast transformation, grayscale transformation, and scale transformation. Each data enhancement processing can enrich the training sample set at a ratio of 1:10 to obtain the purpose of enriching the data sample.

[0072] Step S33: adding a food label and a food freshness label to each third food image to obtain training images.

[0073] The device can annotate the third food image through image processing software, thereby adding food labels and food freshness labels to the third food image to obtain various training images.

[0074] In the technical solution provided in this embodiment, the device performs enhancement processing on food images, thereby enriching the training images and enhancing the recognition and detection robustness of the model.

[0075] In one embodiment, during the model training process, in order to obtain richer feature information, the ImageNet dataset may be pre-trained, that is, a model is trained using labeled food images, and then the model parameters of the trained model are used as the initial training parameters of the multi-task model to speed up the training of the multi-task model.

[0076] In this regard, the device sets initial training parameters for the multi-task model. The initial training parameters are model parameters of the preset model that has completed training. The device then performs multiple rounds of training on the first gradient value and the second gradient value of the multi-task model with the initial training parameters according to the food labels and food freshness labels of each training image, thereby obtaining a recognition model.

[0077] The present invention also provides a food detection device, referring to Figure 5 , the food detection device 500 comprises:

[0078] An input module 510, configured to input a first food image to be detected into a recognition model, wherein the training of the recognition model includes the training of a first gradient value for a food recognition task and the training of a second gradient value for a food freshness recognition task, and the first gradient value and the second gradient value satisfy a gradient normalization condition during the training process;

[0079] The determination module 520 is used to determine the food in the first food image and the freshness of the food according to the output result of the recognition model.

[0080] In one embodiment, the food detection device 500 comprises:

[0081] An acquisition module, used to acquire each training image, wherein the training image includes a food label and a food freshness label;

[0082] The training module is used to perform multiple rounds of training on the first gradient value and the second gradient value of the multi-task model according to the food labels and food freshness labels of each training image to obtain a recognition model, wherein the first gradient value, the first training weight corresponding to the first gradient value, the second gradient value, and the second training weight corresponding to the second gradient value in each round meet the gradient normalization condition.

[0083] In one embodiment, the food detection device 500 comprises:

[0084] An acquisition module, used for acquiring each second food image;

[0085] a processing module, configured to perform data enhancement processing on each second food image to obtain each third food image, wherein the enhancement processing includes contrast transformation, grayscale change and / or scale transformation;

[0086] The adding module is used to add a food label and a food freshness label to each third food image to obtain training images.

[0087] In one embodiment, the food detection device 500 comprises:

[0088] A setting module, used to set initial training parameters for the multi-task model, where the initial training parameters are model parameters of the preset model that has completed training;

[0089] The training module is used to perform multiple rounds of training on the first gradient value and the second gradient value of the multi-task model for setting initial training parameters according to the food label and the food freshness label of each training image.

[0090] Figure 6 is a hardware structure diagram of a food detection device according to an exemplary embodiment.

[0091] The food detection device 600 may include: a processor 61, such as a CPU, a memory 62, and a transceiver 63. Those skilled in the art will appreciate that Figure 6 The structure shown in the figure does not constitute a limitation of the food detection device, and may include more or less components than shown in the figure, or combine certain components, or arrange the components differently. The memory 62 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk.

[0092] The processor 61 can call the computer program stored in the memory 62 to complete all or part of the steps of the above-mentioned food detection method.

[0093] The transceiver 63 is used to receive information sent by an external device and send information to the external device.

[0094] A non-transitory computer-readable storage medium, when the instructions in the storage medium are executed by a processor of a food testing device, enables the food testing device to perform the above-mentioned food testing method.

[0095] A computer program product comprises a computer program, which enables the food testing device to perform the above-mentioned food testing method when the computer program is executed by a processor of the food testing device.

[0096] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. The present invention is intended to cover any variations, uses or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary techniques in the art that are not disclosed in the present disclosure. The description and examples are to be regarded as exemplary only, and the true scope and spirit of the present disclosure are indicated by the following claims.

[0097] It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims

1. A food detection method, characterized in that: include: Inputting a first food image to be detected into a recognition model, wherein the training of the recognition model includes the training of a first gradient value of a food recognition task and the training of a second gradient value of a food freshness recognition task, and the first gradient value and the second gradient value satisfy a gradient normalization condition during the training process; the gradient normalization condition is: the product of the first gradient value and a first training weight plus the product of the second gradient value and a second training weight is equal to a normalized gradient value; the first training weight is determined according to a learning rate of the food recognition task, and the second training weight is determined according to a learning rate of the food freshness task; Determining the food in the first food image and the freshness of the food according to the output result of the recognition model; Before the step of inputting the food image to be detected into the recognition model, the method further includes: Acquire each training image, wherein the training image includes a food label and a food freshness label; According to the food labels and food freshness labels of each of the training images, multiple rounds of training are performed on the first gradient value and the second gradient value of the multi-task model to obtain a recognition model, wherein the first gradient value, the first training weight corresponding to the first gradient value, the second gradient value, and the second training weight corresponding to the second gradient value in each round meet a gradient normalization condition; After the first gradient value and the second gradient value of the multi-task model are trained, the loss function of food recognition and the loss function of food freshness are determined; wherein the loss function of food recognition is determined based on the difference between the first gradient value and the normalized gradient value, and the loss function of food freshness is determined based on the difference between the second gradient value and the normalized gradient value.

2. The food detection method according to claim 1, characterized in that: The step of obtaining each training image comprises: acquiring each second food image; Performing data enhancement processing on each of the second food images to obtain each of the third food images, wherein the enhancement processing includes contrast transformation, grayscale change and / or scale transformation; A food label and a food freshness label are added to each of the third food images to obtain each of the training images.

3. The food detection method according to claim 1, characterized in that: The step of performing multiple rounds of training on the first gradient value and the second gradient value of the multi-task model according to the food label and the food freshness label of each training image to obtain the recognition model comprises: Setting initial training parameters for the multi-task model, wherein the initial training parameters are model parameters of a preset model that has completed training; According to the food labels and food freshness labels of each of the training images, multiple rounds of training are performed on the first gradient value and the second gradient value of the multi-task model for setting initial training parameters.

4. The food detection method according to claim 1, characterized in that: The first training weight is determined according to the learning rate of the food recognition task, and the second training weight is determined according to the learning rate of the food freshness recognition task.

5. The food detection method according to claim 1, characterized in that: The multi-task model includes a first decoder and a second decoder arranged in parallel, each of the first decoder and the second decoder includes three fully connected layers and one output layer, the first decoder is used to identify the food in the food image, and the second decoder is used to identify the freshness of the food in the food image.

6. A food detection device, characterized in that: include: An input module, used for inputting a first food image to be detected into a recognition model, wherein the training of the recognition model includes the training of a first gradient value of a food recognition task and the training of a second gradient value of a food freshness recognition task, and the first gradient value and the second gradient value satisfy a gradient normalization condition during the training process; the gradient normalization condition is: the product of the first gradient value and a first training weight plus the product of the second gradient value and a second training weight is equal to a normalized gradient value; the first training weight is determined according to a learning rate of the food recognition task, and the second training weight is determined according to a learning rate of the food freshness task; a determination module, configured to determine the food in the first food image and the freshness of the food according to the output result of the recognition model; An acquisition module, used to acquire each training image, wherein the training image includes a food label and a food freshness label; According to the food labels and food freshness labels of each of the training images, multiple rounds of training are performed on the first gradient value and the second gradient value of the multi-task model to obtain a recognition model, wherein the first gradient value, the first training weight corresponding to the first gradient value, the second gradient value, and the second training weight corresponding to the second gradient value in each round meet a gradient normalization condition; After the first gradient value and the second gradient value of the multi-task model are trained, the loss function of food recognition and the loss function of food freshness are determined; wherein the loss function of food recognition is determined based on the difference between the first gradient value and the normalized gradient value, and the loss function of food freshness is determined based on the difference between the second gradient value and the normalized gradient value.

7. A food testing device, characterized in that: include: Memory and processor; The memory stores computer-executable instructions; The processor executes the computer-executable instructions stored in the memory, so that the processor performs the food detection method according to any one of claims 1 to 5.

8. The food testing device according to claim 7, characterized in that: The food detection device is a refrigerator.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions, which, when executed by a processor, are used to implement the food detection method according to any one of claims 1 to 5.

10. A computer program product, comprising a computer program, which, when executed by a processor, implements the method according to any one of claims 1 to 5.

Images