Food detection method and device based on neural network model

Abstract

The present application provides a food detection method and device based on a neural network model, which relates to the field of artificial intelligence. The method includes: obtaining a target image of fresh food to be detected; performing convolution processing on the target image to obtain global features and local features of the target image; splicing the global features and local features of the target image to obtain the target image features of the target image; inputting the target image features into the target food quality detection model for processing to obtain the first initial quality information of the fresh food; sending a query request information to a cloud management system to obtain the second initial quality information of the fresh food to be detected, the second initial quality information including the food origin and quality risk of the fresh food to be detected; generating the final quality information of the fresh food to be detected based on the first initial quality information and the second initial quality information.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and more specifically, to a food detection method and device based on a neural network model. Background Art

[0002] With the development of economy and the improvement of living standards, people have higher and higher requirements on food materials, especially fresh food. Such products are not easy to preserve and have a short shelf life. If they are consumed after the expiration date, they may cause physical discomfort. In addition, various safety incidents occur frequently in the food field. When purchasing fresh food, they often encounter quality risks such as mold or chemical treatment such as waxing or sulfur fumigation, which casts an indelible shadow in the minds of consumers.

[0003] Traditional food quality testing methods require special testing equipment for testing. These devices are relatively large and inconvenient to carry. It is often necessary to take relevant food samples to a special place for testing, which makes the testing less convenient. Summary of the invention

[0004] The present application provides a food detection method and device based on a neural network model to detect the quality of fresh food more conveniently and accurately.

[0005] In a first aspect, a food detection method based on a neural network model is provided, the method being executed by a terminal device, the method comprising:

[0006] Determine the search space and the construction unit according to the memory resources of the terminal device, wherein the construction unit is a neural network structural unit obtained by connecting basic operations in the neural network model, and the construction unit is used to build the neural network model;

[0007] Stacking multiple building blocks to get a search network;

[0008] Optimizing the network structure of the construction unit in the search network within the search space to obtain an optimized target construction unit, wherein, in the process of optimizing the network structure of the construction unit in the search network, the search space gradually becomes smaller, and the number of construction units gradually increases, and the reduction of the search space and the increase of the number of construction units make the memory resource occupation value of the terminal device less than a preset first memory threshold value in the process of optimizing the network structure of the construction unit in the search network;

[0009] Building an initial food quality detection model based on the target building unit, the initial food detection model is a neural network model;

[0010] Training an initial food quality detection model based on a training image set to obtain a target food quality detection model after training, wherein the training image set includes a plurality of fresh food images and labeling information corresponding to the plurality of fresh food images, wherein the labeling information corresponding to each fresh food image includes the origin of the fresh food, the type of food, the quality risk, and the probability corresponding to the quality risk;

[0011] In response to a first shooting operation of the user, taking a photo of the fresh food to be inspected to obtain an initial image of the fresh food to be inspected;

[0012] Determine the brightness value of the initial image;

[0013] When the brightness value of the initial image is less than a preset first brightness value, presenting first prompt information on the display interface of the terminal device, the first prompt information being used to prompt the user to retake the image;

[0014] After receiving the user's re-shooting operation, presenting second prompt information on the display interface of the terminal device, the second prompt information is used to prompt the user to increase the brightness value to the first brightness value or above the first brightness value;

[0015] Receiving a brightness adjustment operation from a user, where the brightness adjustment operation is used to adjust the brightness of an image captured by the terminal device to a first brightness value or above the first brightness value;

[0016] In response to a second shooting operation of the user, the fresh food is re-shot when the brightness value reaches or exceeds the first brightness value to obtain a target image of the fresh food to be detected;

[0017] Perform convolution processing on the target image to obtain the global features of the target image;

[0018] Dividing the target image into M sub-images according to the texture layout of the target image, where M is a positive integer greater than 1;

[0019] Perform convolution processing on the M sub-images respectively to extract the image features of the M sub-images and obtain M local features of the target image;

[0020] The global features of the target image and the M local features are concatenated to obtain the target image features of the target image;

[0021] Inputting the target image features into the target food quality detection model for processing to obtain first initial quality information of the fresh food, the first initial quality information including the food origin, food type, quality risk and probability corresponding to the quality risk of the fresh food to be detected;

[0022] Sending a query request message to a cloud server to obtain second initial quality information of the fresh food to be tested, the second initial quality information including the food origin, food type, quality risk and probability corresponding to the quality risk of the fresh food to be tested;

[0023] The final quality information of the fresh food to be tested is generated according to the first initial quality information and the second initial quality information.

[0024] In the embodiment of the present application, in the process of optimizing the construction unit, by gradually reducing the search space and gradually increasing the number of construction units, the occupation of memory resources can be reduced as much as possible while ensuring performance, which is of great significance to the terminal device. In addition, in the present application, the terminal device can obtain the image of the fresh food to be detected in real time by taking pictures, which is convenient for flexible detection. The present application also limits the brightness value of the image so that the brightness value of the image of the fresh food detected by the shooting meets the requirements, which can improve the detection effect to a certain extent. In addition, in the present application, the quality information of the fresh food detected obtained by processing the comprehensive target food quality detection model and the quality information of the fresh food detected obtained from the cloud query can more comprehensively and accurately determine the quality information of the fresh food to be detected. In short, the embodiment of the present application can detect the quality of fresh food more conveniently and accurately.

[0025] In combination with the first aspect, in certain implementations of the first aspect, the initial food quality detection model is trained based on the training image set to obtain a target food quality detection model after training, including:

[0026] Input each training image into the initial food quality detection model to obtain prediction information of the training image;

[0027] Determining a function value of a loss function according to a difference between the prediction information of the training image and the labeling information of the training image;

[0028] The model parameters of the initial food quality detection model are adjusted according to the function value of the loss function until the function value of the loss function is less than the preset target function value to obtain the target food quality detection model.

[0029] In combination with the first aspect, in some implementations of the first aspect, the method further includes:

[0030] When the brightness value of the initial image is greater than the first brightness value, the initial image is determined as the target image.

[0031] When the brightness value of the initial image meets the condition, the initial image can be directly determined as the target image without reshooting the image.

[0032] In combination with the first aspect, in certain implementations of the first aspect, the above-mentioned quality risk types include rot, mildew, lesions, color change, deformation and quality expiration date.

[0033] In combination with the first aspect, in some implementations of the first aspect, the performing convolution processing on the target image to obtain the global features of the target image includes:

[0034] Performing noise reduction processing on the target image to obtain a noise-reduced image;

[0035] The denoised image is convolved to obtain the global features of the target image.

[0036] In the embodiment of the present application, common noise reduction methods can be used to reduce the noise of the image, for example, Gaussian filtering, median filtering, etc. can be used. Among them, Gaussian filtering uses a template (or convolution, mask) to scan each pixel in the image, and replaces the value of the center pixel of the template with the weighted average gray value of the pixels in the neighborhood determined by the template. Median filtering is a statistical sorting filter that sorts all the pixels in the neighborhood and then takes the median value as the pixel in the center of the neighborhood.

[0037] In combination with the first aspect, in certain implementations of the first aspect, the target food quality detection model is a convolutional neural network model.

[0038] In combination with the first aspect, in certain implementations of the first aspect, the final quality information of the fresh food to be tested includes the food origin, food type, quality risk and probability corresponding to the quality risk of the fresh food to be tested.

[0039] In combination with the first aspect, in certain implementations of the first aspect, the method further includes: presenting final quality information of the fresh food to be tested on a display interface of the terminal device.

[0040] In a second aspect, a food detection device is provided, which includes: a memory for storing programs; a processor for executing the programs stored in the memory, and when the program stored in the memory is executed, the processor is used to execute the method in any one of the implementation modes in the first aspect.

[0041] In a third aspect, a computer-readable medium is provided, wherein the computer-readable medium stores a program for execution by a device.

[0042] A program code comprising a method for executing any one of the implementation modes of the first aspect to the fifth aspect. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] Figure 1is a schematic flow chart of a food detection method based on a neural network model according to an embodiment of the present application;

[0044] Figure 2 is a schematic diagram of the building block;

[0045] Figure 3 It is a schematic diagram of the hardware structure of the food quality detection device according to an embodiment of the present application. DETAILED DESCRIPTION

[0046] The following will describe the technical solutions in the embodiments of the present application in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0047] Since the embodiments of the present application involve the application of a large number of neural networks, in order to facilitate understanding, the relevant terms and concepts of the neural networks that may be involved in the embodiments of the present application are first introduced below.

[0048] (1) Neural Network

[0049] Artificial Neural Networks (ANNs), also referred to as Neural Networks (NNs) or Connection Model, is an algorithmic mathematical model that imitates the behavioral characteristics of animal neural networks and performs distributed parallel information processing. This network relies on the complexity of the system and adjusts the interconnected relationships between a large number of internal nodes to achieve the purpose of processing information.

[0050] (2) Deep Neural Networks

[0051] A deep neural network (DNN), also known as a multi-layer neural network, can be understood as a neural network with multiple hidden layers. According to the position of different layers, the neural network inside the DNN can be divided into three categories: input layer, hidden layer, and output layer. Generally speaking, the first layer is the input layer, the last layer is the output layer, and the layers in between are all hidden layers. The layers are fully connected, that is, any neuron in the i-th layer must be connected to any neuron in the i+1-th layer.

[0052] (3) Convolutional Neural Network

[0053] Convolutional neural network (CNN) is a deep neural network with a convolutional structure. Convolutional neural network contains a feature extractor consisting of a convolution layer and a subsampling layer, which can be regarded as a filter. Convolutional layer refers to the neuron layer in the convolutional neural network that performs convolution processing on the input signal. In the convolutional layer of the convolutional neural network, a neuron can only be connected to some neurons in the adjacent layers. A convolutional layer usually contains several feature planes, each of which can be composed of some rectangularly arranged neural units. The neural units in the same feature plane share weights, and the shared weights here are convolution kernels. Shared weights can be understood as the way to extract image information is independent of position. The convolution kernel can be initialized in the form of a matrix of random size, and the convolution kernel can obtain reasonable weights through learning during the training process of the convolutional neural network. In addition, the direct benefit of shared weights is to reduce the connection between the layers of the convolutional neural network, while reducing the risk of overfitting.

[0054] (4) Recurrent neural networks (RNN) are used to process sequence data. In traditional neural network models, the layers are fully connected from the input layer to the hidden layer and then to the output layer, while the nodes within each layer are disconnected. Although this ordinary neural network has solved many difficult problems, it is still powerless to solve many problems. For example, if you want to predict the next word in a sentence, you generally need to use the previous word, because the previous and next words in a sentence are not independent. The reason why RNN is called a recurrent neural network is that the current output of a sequence is also related to the previous output. The specific manifestation is that the network will remember the previous information and apply it to the calculation of the current output, that is, the nodes between the hidden layers are no longer disconnected but connected, and the input of the hidden layer includes not only the output of the input layer but also the output of the hidden layer at the previous moment. In theory, RNN can process sequence data of any length. The training of RNN is the same as the training of traditional CNN or DNN.

[0055] The food detection method based on the neural network model of the embodiment of the present application can be executed by a terminal device, and the terminal device can be a mobile phone, a tablet computer, a PC, a wearable device, a vehicle-mounted device, an augmented reality (AR) / virtual reality (VR) device, an ultra-mobile personal computer (UMPC), a netbook, a personal digital assistant (PDA), and other devices with a camera function. The embodiment of the present application does not impose any restrictions on the specific type of the terminal device.

[0056] Traditional food quality detection methods require special detection equipment for detection. These devices are relatively large and inconvenient to carry. It is often necessary to take the relevant food samples to a special place for detection, and the convenience of detection is poor. In order to solve this problem, it is necessary to develop a portable detection device, by which the quality of fresh food can be quickly and accurately detected. To this end, the embodiment of the present application proposes a food detection method and device based on a neural network model, by building and training a food quality detection model on a terminal device, and then taking photos of the fresh food to be detected, and then processing the photos of the fresh food to be detected by the model, so as to obtain the food quality of the fresh food to be detected. The food detection method based on the neural network model of the embodiment of the present application is described in detail below in conjunction with the accompanying drawings.

[0057] Figure 1 It is a schematic flow chart of the food detection method based on the neural network model in an embodiment of the present application. Figure 1 The method shown can be executed by a terminal device. Figure 1 The method shown includes steps 101 to 107, which are described in detail below.

[0058] 101. Build an initial food quality testing model.

[0059] The construction of the initial food quality detection model in step 101 may specifically include the following steps:

[0060] 101a. Determine a search space and a construction unit according to memory resources of the terminal device.

[0061] Among them, the above-mentioned building unit is a neural network structural unit obtained by connecting basic operations in the neural network model. The building unit is a basic unit or structure used to build a neural network model.

[0062] 101b. Stack multiple building blocks to obtain a search network.

[0063] After obtaining the search network, the building blocks can be further optimized in the search network, and finally an initial food quality detection model can be constructed based on the optimized building blocks.

[0064] 101c. Optimizing the network structure of the building blocks in the search network in the search space to obtain an optimized target building block.

[0065] Among them, in the process of optimizing the network structure of the construction units in the search network, the search space gradually becomes smaller and the number of construction units gradually increases. The reduction of the search space and the increase in the number of construction units make the memory resource occupancy value of the terminal device less than the preset first memory threshold in the process of optimizing the network structure of the construction units in the search network.

[0066] In the process of optimizing the construction units, the search space is gradually reduced and the number of construction units is gradually increased, which can reduce the occupation of memory resources as much as possible while ensuring performance. The above-mentioned first memory threshold can be a value set according to experience. For example, the above-mentioned first memory threshold can be 500M, 1G or other values. The size of the first memory threshold is not limited in the embodiment of the present application.

[0067] Combine the following Figure 2 The building blocks are described in detail.

[0068] like Figure 2 As shown in the figure, the three nodes (node ​​0, node 1 and node 2) in the dotted box constitute a construction unit, which can receive the data output by nodes c_{k-2} and c_{k-1} (c_{k-2} and c_{k-1} can also be feature maps that meet the requirements. For example, c_{k-2} and c_{k-1} can be feature maps obtained after a certain convolution process of the input image), and nodes 0 and 1 process the input data respectively. The data output by node 0 will also be input to

[0069] The data is processed in node 1, and the data output by nodes 0 and 1 is sent to node 2 for processing, and node 2 finally outputs the data processed by the construction unit. In addition, the above nodes c_{k-2} and c_{k-1} can be regarded as input nodes. These two nodes input the data to be processed to the construction unit, and inside the construction unit, 0 and 1 are intermediate nodes, and node 2 is the output node.

[0070] like Figure 2 As shown, the thick arrows represent one or more basic operations, and the basic operation results that converge into the same intermediate node are added at the intermediate node. Figure 2The thin arrows in the figure represent the feature map connection in the channel dimension. The feature map output by output node 2 is formed by connecting the outputs of two intermediate nodes (node ​​0 and node 1) in sequence in the feature map channel dimension.

[0071] The above search space may include pre-set basic operations or combinations of basic operations in a convolutional neural network. These basic operations or combinations of basic operations may be collectively referred to as basic operations.

[0072] The above search space can include the following 8 basic operations:

[0073] (1) Average pooling with a pooling kernel size of 3× 3 (avg_pool_3x3);

[0074] (2) Maximum pooling with a pooling kernel size of 3× 3 (max_pool_3x3);

[0075] (3) Separate convolution with kernel size of 3× 3 (sep_conv_3x3);

[0076] (4) Separate convolution with kernel size of 5×5 (sep_conv_5x5);

[0077] (5) Dilated convolution with kernel size of 3×3 and dilation rate of 2 (dil_conv_3x3);

[0078] (6) Dilated convolution with kernel size of 5×5 and dilation rate of 2 (dil_conv_5x5);

[0079] (7) Skip connection operation;

[0080] (8) Zero operation (Zero, all neurons at the corresponding position are set to zero).

[0081] 101d. Build an initial food quality detection model based on the target construction unit. The initial food detection model is a neural network model.

[0082] In an embodiment of the present application, through the above steps 101a to 101d, the final construction unit can be optimized in the search space while ensuring that memory resources are not occupied too much, so that the initial food quality detection model can be constructed without occupying too many memory resources.

[0083] 102. Train the initial food quality detection model to obtain the target food quality detection model.

[0084] Specifically, in step 102, the initial food quality detection model may be trained based on the training image set to obtain a trained target food quality detection model.

[0085] The training image set includes multiple images of fresh food and labeling information corresponding to the multiple images of fresh food, wherein the labeling information corresponding to each image of fresh food includes the origin of the fresh food, the type of food, the quality risk and the probability corresponding to the quality risk.

[0086] It should be understood that when constructing the above-mentioned training image set, some fresh foods whose origin, food type, quality risk and the probability corresponding to the quality risk are known can be selected, and then the images of these fresh foods can be obtained by taking pictures, and the origin, food type, quality risk and the probability corresponding to the quality risk of these fresh foods can be used as marking information.

[0087] The number of training images in the training image set can be set according to the needs. When a better model needs to be trained, the number of training images can be increased. Otherwise, the number of training images can be appropriately reduced. For example, the training image set includes 10,000 training images and the labeling information corresponding to these training images.

[0088] The following takes the terminal device as an example to describe the model training process in step 102 in detail. Step 102 specifically includes:

[0089] 102a. Input each training image into the initial food quality detection model to obtain prediction information of the training image.

[0090] Specifically, after each training image is input into the initial food quality detection model, the food origin, food type, quality risk and the probability corresponding to the quality risk of the training image can be obtained.

[0091] 102b. Determine a function value of the loss function according to the difference between the prediction information of the training image and the labeling information of the training image.

[0092] In the above step 102b, the prediction information of the training image includes the food origin, food type, quality risk and the probability corresponding to the quality risk of the training image, and the marking information of the training image includes the known food origin, known food type, known quality risk and the known probability corresponding to the quality risk of the training image. The sum of the differences obtained by subtracting the corresponding information in the prediction information of the training image and the marking information of the training image can be used as the function value of the loss function.

[0093] 102c. Adjust the model parameters of the initial food quality detection model according to the function value of the loss function until the function value of the loss function is less than the preset target function value, so as to obtain the target food quality detection model.

[0094] In the above step 102c, when the function value of the loss function does not reach the target function value, the model parameter value of the initial food quality detection model can be adjusted, and then the training image is input into the initial food quality detection model after the modified parameters, and the above steps 102a-102c are repeated again until the function value of the loss function is less than the preset target function value. At this time, the initial food quality detection model has been trained and can be used as the target food quality detection model.

[0095] In the embodiment of the present application, adjusting the model parameters of the initial food quality detection model may specifically be adjusting the number of convolutional layers and the size of the convolution kernel of the initial food quality detection model.

[0096] 103. Obtain a target image of the fresh food to be inspected.

[0097] In the embodiment of the present application, the terminal device can obtain the image of the fresh food to be detected in real time by taking a photo. The above step 103 specifically includes the following steps:

[0098] 103a. In response to a first shooting operation of the user, take a photo of the fresh food to be inspected to obtain an initial image of the fresh food to be inspected.

[0099] 103b. Determine the brightness value of the initial image.

[0100] In the above steps, the brightness value of the initial image can be calculated by the following formula (1).

[0101] L=R*0.299+G*0.587+B*0.114 (1)

[0102] Wherein, L represents the brightness value of the initial image, and R, G, and B are the values ​​of the three components of the initial image respectively.

[0103] 103c. When the brightness value of the initial image is less than a preset first brightness value, a first prompt message is presented on a display interface of the terminal device, where the first prompt message is used to prompt the user to retake the image.

[0104] The first brightness value may be set as required. For example, the value of the first brightness value may be 100. When the brightness value of the initial image is less than 100, it is necessary to re-shoot, and when the brightness value of the initial image is greater than or equal to 100, it is not necessary to re-shoot. It should be understood that 100 here is only an example, and the embodiment of the present application does not limit the specific value of the first brightness value.

[0105] 103d. After receiving the user's re-shooting operation, present a second prompt message on the display interface of the terminal device, where the second prompt message is used to prompt the user to increase the brightness value to the first brightness value or above the first brightness value.

[0106] 103e. Receive a brightness adjustment operation from the user, where the brightness adjustment operation is used to adjust the brightness value when the terminal device captures an image to a first brightness value or above the first brightness value.

[0107] 103f. In response to a second shooting operation of the user, the fresh food is re-shot when the brightness value reaches or exceeds the first brightness value to obtain a target image of the fresh food to be detected.

[0108] When the brightness value of the initial image is too small, it will affect the subsequent image recognition. Therefore, when the brightness value of the initial image is small, the terminal device will prompt the user to retake the image to obtain a target image with a brightness value that meets the requirements, so as to facilitate the subsequent analysis of the fresh food to be inspected based on the target image.

[0109] 104. Input the target image features of the target image of the fresh food to be inspected into the target food quality inspection model for processing to obtain first initial quality information of the fresh food to be inspected.

[0110] The above step 104 specifically includes the following steps:

[0111] 104a. Perform convolution processing on the target image to obtain global features of the target image.

[0112] 104b, dividing the target image into M sub-images according to the texture layout of the target image.

[0113] Among them, M is a positive integer greater than 1, and the value of M can be set according to needs. When the value of M is larger, more local features are obtained, and when the value of M is smaller, smaller local features are obtained.

[0114] For example, the above M may be a value between 5 and 10. The specific value of M is not limited in the present embodiment.

[0115] In the above step 104b, dividing the target image according to the texture layout of the image may be dividing areas with similar textures into one sub-image.

[0116] 104c, performing convolution processing on the M sub-images respectively to extract image features of the M sub-images, and obtaining M local features of the target image;

[0117] 104d, concatenating the global features of the target image and the M local features to obtain a target image feature of the target image;

[0118] 104e. Input the target image features into the target food quality detection model for processing to obtain first initial quality information of the fresh food, the first initial quality information including the food origin, food type, quality risk and probability corresponding to the quality risk of the fresh food to be detected;

[0119] 105. Send a query request to the cloud server to obtain the second initial quality information of the fresh food to be tested.

[0120] In the above step 105, the above query request may be a request message, which may include the QR code information of the fresh food to be tested. After the fresh food to be tested is produced or packaged, the quality information of the fresh food to be tested will be sent to the cloud server. By sending the QR code information of the fresh food to be tested to the cloud server, the second initial quality information of the fresh food to be tested can be obtained. The second initial quality information is the quality information uploaded to the cloud server in advance, and the second initial quality information includes the food origin, food type, quality risk and the probability corresponding to the quality risk of the fresh food to be tested.

[0121] Optionally, the above-mentioned types of quality risks include at least the following appearance changes:

[0122] Rot, mildew, disease, color change, and deformation.

[0123] Among them, the above-mentioned mildew at least includes mildew caused by too many days of storage, the above-mentioned color change at least includes color change caused by dust impurities, radiation, too many days of storage, use of chemicals and / or abnormal dehydration, and the above-mentioned deformation at least includes changes in appearance caused by the use of hormones.

[0124] 106. Obtain final quality information of the fresh food to be tested according to the first initial quality information and the second initial quality information of the fresh food to be tested.

[0125] In some cases, the accuracy of the second initial quality information pre-uploaded to the cloud server is not high enough. Therefore, in the present application, the final quality information of the fresh food to be tested can be generated based on the first initial quality information and the second initial quality information, so that more accurate quality information can be obtained.

[0126] Specifically, in step 106, the food origin, food type, quality risk and probability corresponding to the quality risk of the fresh food to be tested in the first initial quality information can be compared with the food origin, food type, quality risk and probability corresponding to the quality risk of the fresh food to be tested in the second initial quality information. When the food origin and food type of the fresh food to be tested in the first initial quality information are the same as the food origin and food type of the fresh food to be tested in the second initial quality information, the probability corresponding to the quality risk of the fresh food to be tested in the first initial quality information and the probability corresponding to the quality risk of the fresh food to be tested in the second initial quality information are averaged, and the probability corresponding to the quality risk of the fresh food to be tested obtained by averaging is used as the probability corresponding to the quality risk of the fresh food to be tested in the final quality information.

[0127] 107. The final quality information of the fresh food to be tested is presented on the display interface of the smart terminal.

[0128] In step 107, the final quality information of the tested fresh food may be presented on the terminal device interface for the convenience of the user.

[0129] Figure 3 It is a schematic diagram of the hardware structure of the food quality detection device according to an embodiment of the present application. Figure 3 The food quality detection device 2000 shown includes a memory 2001, a processor 2002, a communication interface 2003 and a bus 2004. The memory 2001, the processor 2002 and the communication interface 2003 are connected to each other through the bus 2004.

[0130] The memory 2001 may be a read only memory (ROM), a static storage device, and a random access memory (RAM). The memory 2001 may store a program. When the program stored in the memory 2001 is executed by the processor 2002, the processor 2002 and the communication interface 2003 are used to execute the various steps of the image processing method of the embodiment of the present application.

[0131] Processor 2002 can adopt a general central processing unit (CPU), a microprocessor, an application specific integrated circuit (ASIC), a graphics processing unit (GPU) or one or more integrated circuits to execute relevant programs to implement the functions required to be performed by the units in the food quality detection device of the embodiment of the present application, or to execute the food detection method based on the neural network model of the embodiment of the present application.

[0132] The processor 2002 may also be an integrated circuit chip with signal processing capability. In the implementation process, each step of the image processing method of the embodiment of the present application may be completed by an integrated logic circuit of hardware in the processor 2002 or by instructions in the form of software.

[0133] The processor 2002 may also be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The methods, steps and logic block diagrams disclosed in the embodiments of the present application may be implemented or executed. The general-purpose processor may be a microprocessor or the processor may be any conventional processor, etc. The steps of the method disclosed in the embodiments of the present application may be directly embodied as being executed by a hardware decoding processor, or may be executed by a combination of hardware and software modules in a decoding processor. The software module may be located in a mature storage medium in the art such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory 2001, and the processor 2002 reads the information in the memory 2001, and completes the functions required to be performed by the units included in the food quality detection device of the embodiment of the present application in combination with its hardware, or executes the food detection method based on the neural network model of the embodiment of the present application.

[0134] The communication interface 2003 uses a transceiver device such as, but not limited to, a transceiver to implement communication between the apparatus 4000 and other devices or a communication network. For example, the image to be processed can be acquired through the communication interface 2003 .

[0135] The bus 2004 may include a path for transmitting information between various components of the device 4000 (eg, the memory 2001 , the processor 2002 , and the communication interface 2003 ).

[0136] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0137] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0138] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0139] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0140] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0141] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods described in the various embodiments of the present application. The aforementioned storage media include: various media that can store program codes, such as USB flash drives, mobile hard disks, read-only memories (ROM), random access memories (RAM), magnetic disks or optical disks.

[0142] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A food detection method based on a neural network model, the method is applied to a terminal device, characterized in that: include: Determine the search space and the construction unit according to the memory resources of the terminal device, wherein the construction unit is a neural network structural unit obtained by connecting basic operations in the neural network model, and the construction unit is used to build the neural network model; stacking a plurality of said building blocks to obtain a search network; Optimizing the network structure of the construction unit in the search network within the search space to obtain an optimized target construction unit, wherein, in the process of optimizing the network structure of the construction unit in the search network, the search space gradually becomes smaller, and the number of the construction units gradually increases, and the reduction of the search space and the increase of the number of the construction units make the occupation value of the memory resource of the terminal device less than a preset first memory threshold value in the process of optimizing the network structure of the construction unit in the search network; Building an initial food quality detection model based on the target construction unit, wherein the initial food quality detection model is a neural network model; The initial food quality detection model is trained based on a training image set to obtain a target food quality detection model after training, wherein the training image set includes a plurality of fresh food images and labeling information corresponding to the plurality of fresh food images, wherein the labeling information corresponding to each fresh food image includes the origin of the fresh food, the type of food, the quality risk, and the probability corresponding to the quality risk; In response to a first shooting operation of the user, taking a photo of the fresh food to be inspected to obtain an initial image of the fresh food to be inspected; Determining the brightness value of the initial image; When the brightness value of the initial image is less than a preset first brightness value, presenting first prompt information on the display interface of the terminal device, wherein the first prompt information is used to prompt the user to retake the image; After receiving the re-shooting operation of the user, presenting second prompt information on the display interface of the terminal device, where the second prompt information is used to prompt the user to increase the brightness value to the first brightness value or above the first brightness value; receiving a brightness adjustment operation by the user, wherein the brightness adjustment operation is used to adjust the brightness value of an image captured by the terminal device to the first brightness value or above the first brightness value; In response to the second shooting operation of the user, reshooting the fresh food when the brightness value reaches the first brightness value or is greater than the first brightness value, so as to obtain a target image of the fresh food to be detected; Performing convolution processing on the target image to obtain global features of the target image; Dividing the target image into M sub-images according to the texture layout of the target image, where M is a positive integer greater than 1; Performing convolution processing on the M sub-images respectively to extract image features of the M sub-images to obtain M local features of the target image; Splicing the global feature of the target image and the M local features to obtain a target image feature of the target image; Inputting the target image features into the target food quality detection model for processing to obtain first initial quality information of the fresh food, wherein the first initial quality information includes the food origin, food type, quality risk and probability corresponding to the quality risk of the fresh food to be detected; Sending a query request message to a cloud server to obtain second initial quality information of the fresh food to be tested, wherein the second initial quality information includes the food origin, food type, quality risk and probability corresponding to the quality risk of the fresh food to be tested; generating final quality information of the fresh food to be tested according to the first initial quality information and the second initial quality information; The step of generating the final quality information of the fresh food to be tested according to the first initial quality information and the second initial quality information includes: The food origin, food type, quality risk and probability corresponding to the quality risk of the fresh food to be tested in the first initial quality information are compared with the food origin, food type, quality risk and probability corresponding to the quality risk of the fresh food to be tested in the second initial quality information. When the food origin and food type of the fresh food to be tested in the first initial quality information are the same as the food origin and food type of the fresh food to be tested in the second initial quality information, the probability corresponding to the quality risk of the fresh food to be tested in the first initial quality information and the probability corresponding to the quality risk of the fresh food to be tested in the second initial quality information are averaged, and the probability corresponding to the quality risk of the fresh food to be tested obtained by averaging is used as the probability corresponding to the quality risk of the fresh food to be tested in the final quality information.

2. The method according to claim 1, characterized in that The initial food quality detection model is trained based on the training image set to obtain a target food quality detection model after training, including: Inputting each training image into the initial food quality detection model to obtain prediction information of the training image; Determining a function value of a loss function according to a difference between the prediction information of the training image and the labeling information of the training image; The model parameters of the initial food quality detection model are adjusted according to the function value of the loss function until the function value of the loss function is less than a preset target function value, so as to obtain the target food quality detection model.

3. The method according to claim 1, characterized in that The method further comprises: When the brightness value of the initial image is greater than the first brightness value, the initial image is determined as the target image.

4. The method according to claim 1, characterized in that The types of quality risks include rot, mildew, disease, color change, deformation and quality expiration date.

5. The method according to any one of claims 1 to 4, characterized in that The performing convolution processing on the target image to obtain the global features of the target image includes: Performing noise reduction processing on the target image to obtain a noise-reduced image; The image after the noise reduction process is subjected to convolution process to obtain the global features of the target image.

6. The method according to any one of claims 1 to 4, characterized in that The target food quality detection model is a convolutional neural network model.

7. The method according to any one of claims 1 to 4, characterized in that The final quality information of the fresh food to be tested includes the food origin, food type, quality risk and the probability corresponding to the quality risk of the fresh food to be tested.

8. The method according to any one of claims 1 to 4, characterized in that The method further comprises: The final quality information of the fresh food to be tested is presented on the display interface of the terminal device.

9. A food detection device, characterized in that: include: Memory, used to store programs; A processor is used to execute the program stored in the memory. When the program stored in the memory is executed, the processor is used to perform the following process: Determine the search space and the construction unit according to the memory resources of the terminal device, wherein the construction unit is a neural network structural unit obtained by connecting basic operations in the neural network model, and the construction unit is used to build the neural network model; stacking a plurality of said building blocks to obtain a search network; Optimizing the network structure of the construction unit in the search network within the search space to obtain an optimized target construction unit, wherein, in the process of optimizing the network structure of the construction unit in the search network, the search space gradually becomes smaller, and the number of the construction units gradually increases, and the reduction of the search space and the increase of the number of the construction units make the occupation value of the memory resource of the terminal device less than a preset first memory threshold value in the process of optimizing the network structure of the construction unit in the search network; Building an initial food quality detection model based on the target construction unit, wherein the initial food quality detection model is a neural network model; The initial food quality detection model is trained based on a training image set to obtain a target food quality detection model after training, wherein the training image set includes a plurality of fresh food images and labeling information corresponding to the plurality of fresh food images, wherein the labeling information corresponding to each fresh food image includes the origin of the fresh food, the type of food, the quality risk, and the probability corresponding to the quality risk; In response to a first shooting operation of the user, taking a photo of the fresh food to be inspected to obtain an initial image of the fresh food to be inspected; Determining the brightness value of the initial image; When the brightness value of the initial image is less than a preset first brightness value, presenting first prompt information on the display interface of the terminal device, wherein the first prompt information is used to prompt the user to retake the image; After receiving the re-shooting operation of the user, presenting second prompt information on the display interface of the terminal device, where the second prompt information is used to prompt the user to increase the brightness value to the first brightness value or above the first brightness value; receiving a brightness adjustment operation by the user, wherein the brightness adjustment operation is used to adjust the brightness value of an image captured by the terminal device to the first brightness value or above the first brightness value; In response to the second shooting operation of the user, reshooting the fresh food when the brightness value reaches the first brightness value or is greater than the first brightness value, so as to obtain a target image of the fresh food to be detected; Performing convolution processing on the target image to obtain global features of the target image; Dividing the target image into M sub-images according to the texture layout of the target image, where M is a positive integer greater than 1; Performing convolution processing on the M sub-images respectively to extract image features of the M sub-images to obtain M local features of the target image; Splicing the global feature of the target image and the M local features to obtain a target image feature of the target image; Inputting the target image features into the target food quality detection model for processing to obtain first initial quality information of the fresh food, wherein the first initial quality information includes the food origin, food type, quality risk and probability corresponding to the quality risk of the fresh food to be detected; Sending a query request message to a cloud server to obtain second initial quality information of the fresh food to be tested, wherein the second initial quality information includes the food origin, food type, quality risk and probability corresponding to the quality risk of the fresh food to be tested; generating final quality information of the fresh food to be tested according to the first initial quality information and the second initial quality information; The step of generating the final quality information of the fresh food to be tested according to the first initial quality information and the second initial quality information includes: The food origin, food type, quality risk and probability corresponding to the quality risk of the fresh food to be tested in the first initial quality information are compared with the food origin, food type, quality risk and probability corresponding to the quality risk of the fresh food to be tested in the second initial quality information. When the food origin and food type of the fresh food to be tested in the first initial quality information are the same as the food origin and food type of the fresh food to be tested in the second initial quality information, the probability corresponding to the quality risk of the fresh food to be tested in the first initial quality information and the probability corresponding to the quality risk of the fresh food to be tested in the second initial quality information are averaged, and the probability corresponding to the quality risk of the fresh food to be tested obtained by averaging is used as the probability corresponding to the quality risk of the fresh food to be tested in the final quality information.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium is a program code for execution by an electronic device, and the program code is used to execute the method according to any one of claims 1 to 8.

Images