Method and apparatus for determining freshness, device, and computer program product

Abstract

Embodiments of the present disclosure relate to a method and an apparatus for determining freshness, a device, and a computer program product. The method includes: generating a vision embedding of target food based on image data of the target food by using an image-text alignment model, where the vision embedding indicates image semantic fusion information of the target food. The method further includes: generating a food feature of the target food based on the vision embedding and the image data by using a feature extraction module in a freshness detection model. Therefore, according to the solution provided in the embodiments of the present disclosure, a quantity of categories that the freshness detection model needs to process can be significantly reduced, a training process of the model can be simplified, and an overall freshness detection effect can be improved.

Description

[0001] METHOD AND APPARATUS FOR DETERMINING FRESHNESS, DEVICE, AND COMPUTER PROGRAM PRODUCT

[0002] TECHNICAL FIELD

[0003] Embodiments of the present disclosure relate to the field of computers, and more specifically, to a method and an apparatus for determining freshness, a device, and a computer program product.

[0004] BACKGROUND

[0005] Food freshness detection plays a vital role in food safety and quality control. During storage and transportation, food is prone to impact of environmental factors such as temperature, humidity, and light, causing acceleration of spoilage and deterioration. By means of freshness detection, deteriorated food can be found and processed in time, to prevent a consumer from developing a health problem due to eating unfresh food.

[0006] In recent years, a method for detecting food freshness by using artificial intelligence models has been widely applied to various food industries. The artificial intelligence models can quickly and accurately determine food freshness by learning a large quantity of related information such as food images and smell data. Application of an artificial intelligence technology for food freshness improves food safety and quality, and the technology has become a key technology in the field.

[0007] SUMMARY

[0008] Embodiments of the present disclosure provide a method and an apparatus for determining freshness, a device, a computer program product, and a medium.

[0009] According to a first aspect of the present disclosure, a method for determining freshness is provided. The method includes: generating a vision embedding of target food based on image data of the target food by using an image-text alignment model, where the vision embedding indicates image semantic fusion information of the target food. The method further includes: generating a food feature of the target food based on the vision embedding and the image data by using a feature extraction module in a freshness detection model. In addition, the method further includes: determining freshness of the target food based on the vision embedding and the food feature by using a freshness determining module in the freshness detection model.

[0010] According to a second aspect of the present disclosure, an apparatus for determining freshness is provided. The apparatus includes a vision embedding generation unit, configured to generate a vision embedding of target food based on image data of the target food by using an image-text alignment model, where the vision embedding indicates image semantic fusion information of the target food. The apparatus further includes a food feature generation unit, configured to generate a food feature of the target food based on the vision embedding and the image data by using a feature extraction module in a freshness detection model. In addition, the apparatus further includes a freshness determining unit, configured to determine freshness of the target food based on the vision embedding and the food feature by using a freshness determining module in the freshness detection model.

[0011] According to a third aspect of the present disclosure, an electronic device is provided. The electronic device includes at least one processor and a memory. The memory is coupled to the at least one processor and has instructions stored therein. The instructions, when executed by the at least one processor, cause the device to perform steps of the method according to the first aspect of the present disclosure.

[0012] According to a fourth aspect of the present disclosure, a computer program product is provided. The computer program product is tangibly stored in a non-transitory computer-readable medium and includes computer-executable instructions. The computer-executable instructions, when executed, cause a computer to perform steps of the method of the first aspect of the present disclosure.

[0013] According to a fifth aspect of the present disclosure, a machine-readable storage medium is provided. The machine-readable storage medium has machine-executable instructions stored therein, where the machine-executable instructions are executed by a processor to implement steps of the method according to the first aspect of the present disclosure.

[0014] The summary is for introducing selection of concepts in a simplified form, and the concepts are further described in the following detailed descriptions. The summary is not intended to identify key features or main features of the claimed subject matter, and is not intended to limit the scope of the claimed subject matter.

[0015] BRIEF DESCRIPTION OF THE DRAWINGS

[0016] By describing in more detail exemplary embodiments of the present disclosure with reference to the accompanying drawings, the foregoing and other objects, features, and advantages of the present disclosure will become clearer. In the exemplary embodiments of the present disclosure, the same reference numerals usually represent the same component.

[0017] FIG. l is a schematic diagram of an exemplary environment in which a device and / or a method according to an embodiment of the present disclosure may be implemented;

[0018] FIG. 2 is a flowchart of a method 200 for determining freshness according to an embodiment of the present disclosure;

[0019] FIG. 3 A is a flowchart of a process 300 A of pre-training an image-text alignment model according to an embodiment of the present disclosure;

[0020] FIG. 3B is a schematic diagram of a process of fine-tuning an image-text alignment model according to an embodiment of the present disclosure;

[0021] FIG. 4A is a flowchart of a process of training a freshness detection model according to an embodiment of the present disclosure;

[0022] FIG. 4B is an exemplary structural diagram of an autoencoder model according to an embodiment of the present disclosure;

[0023] FIG. 4C is an exemplary structural diagram of a freshness detection model according to an embodiment of the present disclosure;

[0024] FIG. 5 is a flowchart of a process of determining food freshness according to an embodiment of the present disclosure;

[0025] FIG. 6 is a schematic diagram of an apparatus for determining freshness according to an embodiment of the present disclosure; and

[0026] FIG. 7 is a schematic block diagram of an exemplary device applicable to implementing an embodiment of content of the present disclosure.

[0027] In the accompanying drawings, the same or corresponding numerals indicate the same or corresponding parts. DETAILED DESCRIPTION

[0028] The following describes embodiments of the present disclosure in detail with reference to the accompanying drawings. Although some embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that, the present disclosure may be implemented in various forms and should not be construed as being limited to the embodiments described herein, but the embodiments are provided for a more thorough and complete understanding of the present disclosure. It should be understood that, the accompanying drawings and the embodiments of the present disclosure are merely intended for exemplary purposes, and are not intended to limit the protection scope of the present disclosure.

[0029] In descriptions of the embodiments of the present disclosure, the term "include" and similar terms should be understood as open inclusion, that is, "include but are not limited to". The term "based on" should be understood as "at least partially based on". The term "an embodiment" or "this embodiment" should be understood as "at least one embodiment". The terms "first", "second", and the like may represent different or same objects. Unless explicitly stated, other definitions may be included explicitly or implicitly in the following specification.

[0030] As described above, an artificial intelligence technology is more widely applied to the field of food freshness detection. However, in the related technology, freshness levels of different food are usually regarded as independent categories, and a latent relationship between the categories cannot be fully used. In such a processing manner, training complexity of an artificial intelligence model is increased, and a freshness detection effect is also limited. As food types increase, a quantity of categories that the model needs to process increases correspondingly, causing longer training time and more difficult convergence of the model, and then affecting detection accuracy.

[0031] Therefore, an embodiment of the present disclosure provides a food freshness detection solution. In this solution, an image-text alignment model is introduced to generate a vision embedding of target food by using image data of the target food. Then, a feature of the target food is extracted based on the generated vision embedding and the image data by using a feature extraction module in a freshness detection model. Finally, freshness of the target food is determined based on the vision embedding and the extracted feature by using a freshness determining module in the freshness detection model.

[0032] Therefore, the vision embedding of the target food is generated by using the image-text alignment model. The vision embedding includes a vision feature of the food, and also implicitly includes semantic information associated with the feature. By using the vision embedding, the freshness detection model does not need to separately model each freshness level of each food, and may perform freshness detection by using a unified vision embedding representation. Therefore, according to the solution in this embodiment of the present disclosure, a quantity of categories that the freshness detection model needs to process can be significantly reduced, a training process can be simplified, and an overall food freshness detection effect can be improved.

[0033] The following describes the basic principles and several exemplary implementations of the present disclosure with reference to FIG. 1 to FIG. 7. It should be understood that, the exemplary embodiments are provided only to cause a person skilled in the art to better understand and implement the embodiments of the present disclosure, and are not intended to limit the scope of the present disclosure in any way.

[0034] FIG. 1 shows an exemplary environment 100 in which a device and / or a method according to an embodiment of the present disclosure may be implemented. As shown in FIG. 1, the exemplary environment 100 may include a computing device 110. The computing device 110 may be a user terminal, a mobile device, a computer, or the like, or may be a computing system, a single server, a distributed server, or a cloud-based server. The computing device 110 may obtain a food image 120 of target food (for example, food that needs to be detected). For example, the food image 120 may reflect features such as an appearance, a texture, a color, and a shape of the target food, to provide basic data for subsequent processing and analysis. In some embodiments, high-resolution image data that may be acquired by using a color (RGB) camera or a hyperspectral camera may be used.

[0035] The computing device 110 may include an image-text alignment model 130. The image-text alignment model 130 may determine a vision embedding 140 of the target food according to the food image 120. In some embodiments, the image-text alignment model 130 may be a pre-trained cross-domain image-text alignment model. In some embodiments, fine-tuning may be further performed by using data of a specific domain. For example, a pre-training model is fine-tuned by using image and text data related to food, to generate the image-text alignment model used in a food domain. It should be understood that, a vision embedding of food may alternatively be generated by using another image-text alignment model. This is not limited in the present disclosure.

[0036] The computing device 110 may include a freshness detection model 150. The freshness detection model 150 may include a feature extraction module 160 and a freshness determining module 180. The feature extraction module 160 in the freshness detection model 150 may generate a food feature 170 based on the food image 120 and the vision embedding 140. For example, the food image 120 may be high-dimensional vector data related to the image data, and the food feature 170 may be low-dimensional vector data that has been subjected to dimension reduction. The food feature 170 keeps core information of the food image 120, and also significantly reduces a data dimension, thereby improving efficiency and accuracy of subsequent processing.

[0037] The freshness determining module 180 in the freshness detection model 150 may determine freshness 190 of the target food based on the food feature 170 and the vision embedding 140. For example, the freshness determining module 180 may be a classifier module, and the freshness 190 may be different freshness categories, for example, a fresh category, a half-fresh category, or a spoiled category. The different freshness categories may alternatively be understood as different freshness levels. It should be understood that, this is merely an example herein, and division of the freshness level is not limited in the embodiments of the present disclosure.

[0038] It should be understood that, an architecture and functions of the exemplary environment 100 are described for merely exemplary purposes, and do not imply any limitation on the scope of the present disclosure. The embodiments of the present disclosure may further be applied to another environment having a different structure and / or function.

[0039] The following describes a process according to this embodiment of the present disclosure in detail with reference to FIG. 2 to FIG. 7. For ease of understanding, specific data mentioned in the following descriptions is exemplary, and is not used to limit the protection scope of the present disclosure. It may be understood that, the embodiments described below may further include additional actions that are not shown and / or the shown actions that may be omitted, and the scope of the present disclosure is not limited in this aspect.

[0040] FIG. 2 shows a flowchart of a method 200 for determining freshness according to an embodiment of the present disclosure. At a block 202, a vision embedding of target food may be generated based on image data of the target food by using an image-text alignment model, where the vision embedding indicates image semantic fusion information of the target food. For example, for describing with reference to FIG. 1, the computing device 110 may generate the vision embedding 140 of the target food based on the image data 120 of the target food by using the image-text alignment model 130, where the vision embedding 140 indicates the image semantic fusion information of the target food.

[0041] At a block 204, a food feature of the target food may be generated based on the vision embedding and the image data by using a feature extraction module in a freshness detection model. For example, for describing with reference to FIG. 1, the computing device 110 may generate the food feature 170 of the target food based on the vision embedding 140 and the image data 120 by using the feature extraction module 160 in the freshness detection model 150.

[0042] At a block 206, freshness of the target food may be determined based on the vision embedding and the food feature by using a freshness determining module in the freshness detection model. For example, for describing with reference to FIG. 1, the computing device 110 may determine the freshness 190 of the target food based on the vision embedding 140 and the food feature 170 by using the freshness determining module 180 in the freshness detection model 150.

[0043] Therefore, according to the method 200 in this embodiment of the present disclosure, the vision embedding of the target food is generated by using the image-text alignment model. The vision embedding includes a vision feature of the food, and also implicitly includes semantic information associated with the feature. By using the vision embedding, the freshness detection model does not need to separately model each freshness level of each food, and may perform freshness detection by using a unified vision embedding representation. Therefore, according to the solution in this embodiment of the present disclosure, a quantity of categories that the freshness detection model needs to process can be significantly reduced, a training process can be simplified, and an overall food freshness detection effect can be improved.

[0044] FIG. 3A shows a flowchart of a process 300A of pre-training an image-text alignment model according to an embodiment of the present disclosure. At a block 302, a pre-trained cross-domain image-text alignment model may be obtained. For example, an open source model, for example, a comparison language-image pre-training (CLIP) model, trained by using large-scale data sets may be obtained from a public model base. The pre-training model is usually trained by using large and diversified data sets. The data sets include images from different domains and text descriptions related to the images. In a training process, the model aligns vision and text information in high-dimensional embedding space by using a comparison learning method, so that the model can understand and associate vision content in an image with semantic information in text, to form abundant multi-modal representations.

[0045] For example, during training of the CLIP model, a data set including hundreds of millions of pairs of images and text is used. The images may come from different domains and scenarios, for example, natural scenery, a daily object, and an animal. The text descriptions may cover various semantic information of the images, for example, a name, an attribute, a behavior, an environment background of an object in the images. In some embodiments, the vision embedding may be generated by using the pre-trained cross-domain image-text alignment model, and fine-tuning does not need to be performed for the food domain. The cross-domain model pre-trained on a large scale is obtained, so that a powerful image and text alignment capability can be quickly established in the food domain, and large-scale model training does not need to be performed from the beginning. In this way, computing resources are saved and a development period is also greatly shortened, so that the freshness detection model can be quickly put into actual application.

[0046] At a block 304, image-text alignment training data used in the food domain may be obtained. For example, the image-text alignment training data used in the food domain may include a food image and a text description corresponding to the food image, to cause the model to learn how to associate a vision feature with semantic information. For example, the image data may include various types of fruit, and the fruit may be in different freshness statuses. For example, an image may include very fresh fruit (for example, an apple that is just picked), slightly-half-fresh fruit (for example, a banana that has been stored for a period of time), near-spoiled fruit (for example, an orange that has a speckled surface), or obviously-spoiled fruit (for example, a strawberry that darkens and has a soft texture).

[0047] In some embodiments, the text description part may only include a name of fruit (that is, a fruit category), for example, "an apple", "a banana", or "an orange". In some embodiments, the text description may further include a freshness status of food. For example, the text description may be "a fresh apple", "a slightly-softened banana", "a near-spoiled orange", "a completely-spoiled strawberry", or the like. The text description with a fine granularity helps the model to understand and classify fruit in different statuses more accurately, and also helps the model to achieve higher recognition accuracy in a freshness detection task. It should be understood that, the use of the fruit for description herein is merely intended for exemplary purposes, and types of food is not limited in the embodiments of the present disclosure.

[0048] At a block 306, the obtained image-text alignment model may be fine-tuned based on the image-text alignment training data used in the food domain, to generate an image-text alignment model for the food. For example, the pre-trained cross-domain image-text alignment model is fine-tuned by using a data set including fruit images in various freshness statuses and text descriptions corresponding to the fruit images. The fine-tuning process may make the model further applicable to specific features of the food domain, and in particular, vision and semantic association of food freshness. For example, in the fine-tuning process, the model re-learns how to align a fine vision difference (for example, a color change or a texture change) in the food image with the corresponding text description (for example, "fresh", "half-fresh", or "spoiled"). Through iterative training for a plurality of times, the model gradually adjusts a weight of the model, so that the image-text alignment model can better capture key features of the food in different freshness statuses.

[0049] FIG. 3B shows a schematic diagram of a process 300B of performing fine-tuning on an image-text alignment model according to an embodiment of the present disclosure. As shown in FIG. 3B, in the fine-tuning process, text data 312 and image data 318 may be used. For example, the text data 312 may include text information related to food, such as a name of fruit and a freshness description of the fruit. The text data 312 may be processed by using a text encoder 314. For example, the text encoder 314 may be a model based on a Transformer. An encoded text feature is output to a text feature set 316, which represents text embeddings of different text descriptions (Tl, T2, ..., TN).

[0050] The image data 318 may be a food image, for example, an image of a banana shown in the figure. The image data 318 is processed by using an image encoder 320, to generate a corresponding image feature set 322. Each image embedding matches a specific text embedding, and an image-text alignment mapping relationship is formed in a feature alignment matrix 324. Each element of the feature alignment matrix 324 represents alignment results between specific image embeddings (II, 12, ..., IN) and different text embeddings (Tl, T2, ..., TN), that is, cosine similarities between the image embeddings and the text embeddings. The many-to-many mapping relationship between the images and the text in the fine-tuning process is used, so that the model can gradually adjust the weight, so that a vision embedding and a semantic embedding are more precisely aligned in the food domain. Such alignment can better capture a freshness feature of food, and can also enhance performance of the model in food freshness detection.

[0051] FIG. 4A shows a flowchart of a process 400A of training a freshness detection model according to an embodiment of the present disclosure. At a block 402, an autoencoder model used for food may be trained by using image data of the food. An autoencoder is an unsupervised learning neural network, and reconstructs input data by learning a compression representation of data. The autoencoder may include an encoder and a decoder. The encoder is configured to compress the input data into a low-dimensional representation, and the decoder is configured to restore the representation to original data. FIG. 4B shows a structure of an autoencoder model used for food. The following describes the autoencoder model with reference to FIG. 4B.

[0052] FIG. 4B shows an exemplary structural diagram of an autoencoder model 400B according to an embodiment of the present disclosure. As shown in FIG. 4B, the autoencoder model 400B may include an encoder module 424, a sampling module 430, and a decoder module 434. A combination of the encoder module 424 and the sampling module 430 may be referred to as a feature extraction module. Specifically, the encoder module 424 receives a food image 422 and a vision embedding 420 (for example, a vision embedding generated by using an image-text alignment model), and encodes the food image 422 and the vision embedding 420, to generate a parameter of a feature distribution (for example, a parameter of a multivariate Gaussian distribution), for example, a mean (p) vector parameter 426 and a logarithmic variance (logo2) vector parameter 428. The vision embedding 420 is introduced as a condition variable of the autoencoder model, and is used for providing prior knowledge or context information for the autoencoder model when a latent representation is generated. In some embodiments, the condition variable is the vision embedding of the food image, and the vision embedding is generated by using the image-text alignment model.

[0053] The sampling module 430 may perform a sampling operation by using the parameter of the feature distribution, to generate a latent vector 432 as an extracted food feature for a next decoding process. Subsequently, the decoder module 434 combines the latent vector 432 and the vision embedding 420, to reconstruct output data 436 corresponding to the input data. In such a process, important features of the food image 422 are kept, and adjustment are also performed according to additional information provided by the vision embedding 420, so that the reconstructed output data 436 can better conform to characteristics of the food image 422.

[0054] Therefore, according to this embodiment of the present disclosure, the autoencoder model can efficiently learn and capture a latent distribution of the food image, and accurately generate the food image with reference to a vision embedding in a decoding process. In addition, the autoencoder model can implement unsupervised training, which means that the food image does not need to be manually labeled. This significantly reduces costs of data labeling, and also reduces manual intervention in a training process, thereby improving training efficiency and scalability.

[0055] Referring to FIG. 4A again, the image data of the food at the block 402 may be acquired by using a color (RGB) camera or a hyperspectral camera. For example, high-resolution image data may be acquired by the RGB camera or the hyperspectral camera, and the image data can capture features such as a color, a shape, and a surface texture of the food, and provide abundant vision information for the model to learn and extract important features related to freshness.

[0056] At a block 404, the decoder module in the autoencoder model may be replaced with the classifier module, to obtain the freshness detection model of the food. As described above, the autoencoder is trained, so that the feature extraction module has a capability of extracting the food feature from the image data. Specifically, the autoencoder converts the input data into feature representations in latent space by using the feature extraction module, and the feature representations capture core information of the food image. However, as a generative model, the autoencoder can generate a food image (that is, an output of the decoder) similar to the input data, but cannot directly determine a freshness category of the food. This is because a target of the autoencoder is mainly to reconstruct the input data, rather than to perform a classification task.

[0057] In an application scenario of food freshness detection, to determine the freshness of the food, in this embodiment of the present disclosure, the original decoder module is replaced with the classifier module, and the classifier module may determine the food freshness based on the food feature generated by the feature extraction module. In this way, the freshness detection model keeps a feature extraction capability of the autoencoder, and can also use the extracted feature for classification and determining of a freshness level. FIG. 4C shows an exemplary structure of a freshness detection model. The freshness detection model is described below with reference to FIG. 4C.

[0058] FIG. 4C shows an exemplary structural diagram of a freshness detection model 400C according to an embodiment of the present disclosure. As shown in FIG. 4C, the freshness detection model 400C may include an encoder module 444, a sampling module 450, and a classifier module 454. A combination of the encoder module 444 and the sampling module 450 may be referred to as a feature extraction module. The encoder module 444 may receive a food image 442 and a vision embedding 440, to generate a mean vector parameter 446 and a logarithmic variance vector parameter 448 of the feature distribution.

[0059] The sampling module 450 may generate a food feature 452 (that is, z in FIG. 4C) through a sampling operation based on the distribution parameter generated by the encoder 344. The food feature 452 may represent a core feature of the food image 442. The classifier module 454 receives the food feature 452 generated by the sampling module 450, and classifies food freshness with reference to the vision embedding 440. The classifier module 454 may map the vision embedding 440 and the food feature 452 to a specific freshness category through training, to detect the food freshness. Finally, the classifier module 456 may output an output result 456 representing a food freshness category.

[0060] Referring to FIG. 3A again, at the block 306, the freshness detection model may be trained based on the training data. As described above, the feature extraction module (the encoder module and the sampling module) in the freshness detection model has been trained, and the classifier module in the freshness detection model further needs to be trained. For example, supervised training may be performed on the freshness detection model based on labeled training data. For example, the training data may be a food type corresponding to image data related to food, and a category labeled on the training data may be fresh, half-fresh, or spoiled, which respectively indicate different freshness levels.

[0061] FIG. 5 shows a flowchart of a process 500 of determining food freshness according to an embodiment of the present disclosure. At a block 502, image data related to to-be-detected food may be obtained. In some embodiments, the image data may be timeseries multi-angle image data. For example, photos of the food may be acquired and stored at a predetermined time interval. In some embodiments, the timeseries multi-angle image data may be acquired at a predetermined angle and the predetermined time interval by using at least one of the color camera or the hyperspectral camera. The timeseries multi-angle image data may provide abundant information because the timeseries multi-angle image data can capture changes of the food at different time points and angles, so that the freshness detection model can analyze change trends to determine a freshness attenuation process.

[0062] At a block 504, a vision embedding of the to-be-detected food may be generated by using the image-text alignment model. For example, the food image may be processed by using the pre-trained cross-domain image-text alignment model to generate the vision embedding. In some embodiments, a timeseries multi-angle vision embedding of the target food may be generated based on the timeseries multi-angle image data of the target food by using the image-text alignment model.

[0063] At a block 506, the freshness of the food may be determined based on the vision embedding and the input data. For example, with reference to FIG. 4C, the vision embedding 440 and the food image 342 may be input into the freshness detection model, to determine the freshness category of the food. An encoder in the freshness detection model may determine the parameter of the feature distribution for the food according to the vision embedding and the food image. In some embodiments, a multivariate Gaussian distribution may be used. In this case, an output parameter of the encoder may be a mean vector and a variance vector for the multivariate Gaussian distribution. A sampler in the freshness detection model may perform sampling based on the parameter of the feature distribution, to generate a feature vector of the food. Then, a classifier in the freshness detection model may determine the freshness of the food based on the vision embedding and the feature vector of the food.

[0064] At a block 508, prompt information may be sent to a user based on the freshness of the food. In some embodiments, the category of the freshness may be fresh, half-fresh, or spoiled. For example, when it is detected that food is half-fresh or spoiled, a notification may be sent to the user for processing. In some embodiments, if the output freshness indicates that the target food is in the half-fresh category, prompt information may be sent to the user, to prompt the user that a life of the food to a shelf life is a preset number of days, and the preset number of days may be configured. In some embodiments, if the output freshness indicates that the target food is in the spoiled category, warning information may be sent to the user, to warn the user that the life of the food exceeds the shelf life. In this way, when receiving the prompt information, the user can learn about a situation in which the food is about to approach the shelf life in time, to eat the food or take other processing measures, such as freezing or cooking the food, to avoid a waste caused by expired food. In addition, when receiving the warning information, the user may process the spoiled food, to prevent a health risk or another negative consequence caused by easting by mistake.

[0065] FIG. 6 shows a schematic diagram of an apparatus 600 for determining freshness according to an embodiment of the present disclosure. The apparatus 600 includes a vision embedding generation unit 602, configured to generate a vision embedding of target food based on image data of the target food by using an image-text alignment model, where the vision embedding indicates image semantic fusion information of the target food. The apparatus 600 further includes a food feature generation unit 604, configured to generate a food feature of the target food based on the vision embedding and the image data by using a feature extraction module in a freshness detection model. In addition, the apparatus 600 further includes a freshness determining unit 606, configured to determine the freshness of the target food based on the vision embedding and the food feature by using a freshness determining module in the freshness detection model.

[0066] In some embodiments, the food feature generation unit 604 includes: a feature distribution generation unit, configured to generate a feature distribution of the target food based on the image data and the vision embedding by using an encoder in the feature extraction module; and a second food feature generation unit, configured to generate the food feature based on the feature distribution of the target food by using a sampling module in the feature extraction module.

[0067] In some embodiments, the feature distribution generation unit includes: a distribution parameter generation unit, configured to generate a parameter of the feature distribution based on the image data and the vision embedding by using the encoder; and a second feature distribution generation unit, configured to determine the feature distribution of the target food based on the parameter of the feature distribution.

[0068] In some embodiments, the feature distribution of the target food includes a multivariate Gaussian distribution, and the parameter of the feature distribution includes a mean vector parameter and a variance vector parameter.

[0069] In some embodiments, the image-text alignment model is a pre-trained cross-domain image-text alignment model.

[0070] In some embodiments, the apparatus 600 further includes: a fine-tuning data obtaining unit, configured to obtain fine-tuning data used for fine-tuning the pre-trained cross-domain image-text alignment model, where the fine-tuning data includes a food image and a text description associated with the food image; and an image-text model generation unit, configured to fine-tune the pre-trained cross-domain image-text alignment model by using the fine-tuning data, to generate the image-text alignment model for food.

[0071] In some embodiments, the image data includes timeseries multi-angle image data, and the vision embedding generation unit 602 includes a second vision embedding generation unit, configured to generate a timeseries multi-angle vision embedding of the target food based on the timeseries multi-angle image data of the target food by using the image-text alignment model.

[0072] In some embodiments, the apparatus 600 further includes an image data acquisition unit, configured to acquire the timeseries multi-angle image data at a pre-determined angle and a pre-determined time interval by using at least one of a color camera or a hyperspectral camera.

[0073] In some embodiments, the apparatus 600 further includes: a training data obtaining unit, configured to obtain food training data, where the food training data includes image data of food; and an autoencoder training unit, configured to train an autoencoder model used for the food based on the food training data through unsupervised learning, where the autoencoder model includes the feature extraction module and a decoder module, and the vision embedding generated by using the image-text alignment model is input to the feature extraction module and the decoder module as a condition variable of the autoencoder model.

[0074] In some embodiments, the apparatus 600 further includes: an extraction module obtaining unit, configured to obtain the trained feature extraction module in the trained autoencoder model; a detection model generation unit, configured to generate the freshness detection model based on the trained feature extraction module and the freshness determining module; and a determining module training unit, configured to train the freshness determining module in the freshness detection model based on the food training data and food type data related to the food training data.

[0075] FIG. 7 shows a schematic block diagram of an exemplary device 700 applicable to implementing an embodiment of the present disclosure. As shown in the figure, the device 700 includes a processor 701, which can perform various appropriate actions and processing according to computer program instructions stored in a read-only memory (ROM) 702 or computer program instructions loaded to a random access memory (RAM) 703. Various programs and data required for the operation of the device 700 may further be stored in the RAM 703. The processor 701, the ROM 702, and the RAM 703 are connected to each other through a bus 704. An input / output (I / O) interface 705 is also connected to the bus 704.

[0076] The methods and processes described above may be performed by the processor 701. For example, in some embodiments, the methods and processes described above may be implemented as a computer software program that is tangibly included in a machine-readable medium. In some embodiments, a part or all of the computer program may be loaded and / or installed on the device 700 by using the ROM 702. When the computer program is loaded to the RAM 703 and executed by the processor 701, one or more actions of the foregoing described method and process may be performed.

[0077] The present disclosure may be a method, an apparatus, a system, and / or a computer program product. The computer program product may include a computer-readable storage medium, on which computer-readable program instructions used for performing aspects of the present disclosure are loaded.

[0078] The computer-readable storage medium may be a tangible device that may hold and store instructions used by an instruction execution device. For example, the computer-readable storage medium may be but is not limited to, an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any appropriate combination thereof. In a more specific example (a non-exhaustive list), the computer-readable storage medium includes: a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a static random access memory (SRAM), and any appropriate combination of the above. The computer-readable storage medium used herein is not construed as an instantaneous signal, for example, a radio wave or another freely propagating electromagnetic wave, an electromagnetic wave propagating through a waveguide or another transmission medium (for example, an optical pulse passing through an optical fiber cable), or an electric signal transmitted through an electric wire.

[0079] The computer-readable program instructions described herein may be downloaded to each computing / processing device from a computer-readable storage medium, or may be downloaded to an external computer or external storage device through a network such as the Internet, a local area network, a wide area network and / or a wireless network. The network may include a copper transmission cable, optical fiber transmission, wireless transmission, a router, a firewall, a switch, a gateway computer, and / or an edge server. A network adapter card or a network interface in each computing / processing device receives computer-readable program instructions from the network, and forwards the computer-readable program instructions for storage in the computer-readable storage medium in each computing / processing device.

[0080] The computer program instructions for performing the operations of the present disclosure may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-related instructions, microcode, firmware instructions, state setting data, or source code or target code written in any combination of one or more programming languages. The programming languages include an object-oriented programming language such as Smalltalk or C++, and a conventional procedural programming language such as C programming language or a similar programming language. The computer-readable program instructions may be completely executed on a user computer, partially executed on the user computer, executed as an independent software package, partially executed on the user computer and partially executed on a remote computer, or completely executed on a remote computer or a server. For the case involving a remote computer, the remote computer may be connected to a computer of a user through any type of network including a Local Area Network (LAN) or a Wide Area Network (WAN), or may be connected to an external computer (for example, through the Internet by means of an Internet service provider). In some embodiments, an electronic circuit is personalized and customized through state information of the computer-readable program instructions, for example, a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA). The electronic circuit may execute the computer-readable program instructions, thereby implementing all aspects of the present disclosure.

[0081] All of the aspects of the present disclosure are described herein with reference to the flowcharts and / or the block diagrams of the method, the apparatus (system), and the computer program product in the embodiments of the present disclosure. It should be understood that, each block in the flowcharts and / or the block diagrams and a combination of the blocks in the flowcharts and / or the block diagrams may be both implemented through the computer-readable program instructions.

[0082] The computer-readable program instructions may be provided to a general-purpose computer, a dedicated computer, or a processing unit of another programmable data processing apparatus, to produce a machine. In this way, the instructions, when executed through the computer or the processing unit of the another programmable data processing apparatus, generate an apparatus that implements the functions / actions specified in one or more blocks in the flowcharts and / or the block diagrams. The computer-readable program instructions may alternatively be stored in the computer-readable storage medium. The instructions enable the computer, the programmable data processing apparatus, and / or another device to operate in a specific manner. Therefore, the computer-readable medium having the instructions stored therein includes a manufactured product, which includes instructions for implementing all aspects of the functions / actions specified in one or more blocks in the flowcharts and / or the block diagrams.

[0083] The computer-readable program instructions may alternatively be loaded to the computer, the another programmable data processing apparatus, or the another device, so that a series of operations and steps are performed on the computer, the another programmable data processing apparatus, or the another device, to generate a process of computer implementation. In this way, the instructions executed on the computer, the another programmable data processing apparatus, or the another device implement the functions / actions specified in one or more blocks in the flowcharts and / or the block diagrams.

[0084] The flowcharts and block diagrams in the accompanying drawings show a system architecture, functions, and operations that may be implemented by using the system, the method, and the computer program product according to a plurality of embodiments of the present disclosure. In this regard, each block in the flowcharts or the block diagrams may represent a module, a program segment, or a part of instructions. The module, the program segment, or the part of the instructions includes one or more executable instructions for implementing a specified logic function. In some implementations used as substitutes, functions annotated in the blocks may alternatively occur in a sequence different from that annotated in the accompanying drawings. For example, actually two blocks shown in succession may be performed basically in parallel, and sometimes the two blocks may be performed in a reverse sequence. This depends on a related function. It should also be noted that, each block in the block diagrams and / or the flowcharts and combinations of blocks in the block diagrams and / or the flowcharts may be implemented by using a dedicated hardware-based system configured to perform a specified function or operation, or may be implemented by using a combination of dedicated hardware and computer instructions.

[0085] The embodiments of the present disclosure have been described above, and the above descriptions are exemplary, non-exhaustive, and is not limited to the disclosed embodiments. Without departing from the scope and the spirit of the various embodiments described, many modifications and changes are apparent to a person of ordinary skill in the art. The terms used in this specification are selected to best explain principles of the embodiments, practical application, or improvements to technologies in the market, or enable a person of ordinary skill in the art to understand embodiments disclosed herein.

Claims

CLAIMSWhat is claimed is:

1. A method (200) for determining freshness, characterized by comprising: generating (202) a vision embedding of target food based on image data of the target food by using an image-text alignment model, wherein the vision embedding indicates image semantic fusion information of the target food; generating (204) a food feature of the target food based on the vision embedding and the image data by using a feature extraction module in a freshness detection model; and determining (206) freshness of the target food based on the vision embedding and the food feature by using a freshness determining module in the freshness detection model.

2. The method (200) according to claim 1, characterized in that, the generating a food feature of the target food comprises: generating a feature distribution of the target food based on the image data and the vision embedding by using an encoder in the feature extraction module; and generating the food feature based on the feature distribution of the target food by using a sampling module in the feature extraction module.

3. The method (200) according to any one of claims 1 or 2, characterized in that, the generating a feature distribution of the target food comprises: generating a parameter of the feature distribution based on the image data and the vision embedding by using the encoder; and determining the feature distribution of the target food based on the parameter of the feature distribution.

4. The method (200) according to any of claims 1 to 3, characterized in that, the feature distribution of the target food comprises a multivariate Gaussian distribution, and the parameter of the feature distribution comprises a mean vector parameter and a variance vector parameter.

5. The method (200) according to any of claims 1 to 4, characterized in that, the image-text alignment model is a pre-trained cross-domain image-text alignment model.

6. The method (200) according to claim 5, characterized by further comprising: obtaining fine-tuning data used for fine-tuning the pre-trained cross-domainimage-text alignment model, wherein the fine-tuning data comprises a food image and a text description associated with the food image; and fine-tuning the pre-trained cross-domain image-text alignment model by using the fine-tuning data, to generate the image-text alignment model for food.

7. The method (200) according to any of claims 1 to 6, characterized in that, the image data comprises timeseries multi-angle image data, and the generating a vision embedding of target food comprises: generating a timeseries multi-angle vision embedding of the target food based on the timeseries multi-angle image data of the target food by using the image-text alignment model.

8. The method (200) according to claim 7, characterized by further comprising: acquiring the timeseries multi-angle image data at a pre-determined angle and a pre-determined time interval by using at least one of a color camera or a hyperspectral camera.

9. The method (200) according to claim 8, characterized by further comprising: obtaining food training data, wherein the food training data comprises image data of food; and training an autoencoder model used for the food based on the food training data through unsupervised learning, wherein the autoencoder model comprises the feature extraction module and a decoder module, and the vision embedding generated by using the image-text alignment model is input to the feature extraction module and the decoder module as a condition variable of the autoencoder model.

10. The method (200) according to claim 9, characterized by further comprising: obtaining the trained feature extraction module in the trained autoencoder model; generating the freshness detection model based on the trained feature extraction module and the freshness determining module; and training the freshness determining module in the freshness detection model based on the food training data and food type data related to the food training data.

11. An apparatus for determining freshness, characterized by comprising: a vision embedding generation unit, configured to generate a vision embedding of target food based on image data of the target food by using an image-text alignment model,wherein the vision embedding indicates image semantic fusion information of the target food; a food feature generation unit, configured to generate a food feature of the target food based on the vision embedding and the image data by using a feature extraction module in a freshness detection model; and a freshness determining unit, configured to determine freshness of the target food based on the vision embedding and the food feature by using a freshness determining module in the freshness detection model.

12. An electronic device, characterized by comprising: at least one processor; and a memory, coupled to the at least one processor and having instructions stored therein, wherein the instructions, when executed by the at least one processor, cause the device to perform the method according to any one of claims 1 to 10.

13. A computer program product, tangibly stored in a non-transitory computer-readable medium and comprising machine-executable instructions, wherein the machine-executable instructions are used for performing the method according to any one of claims 1 to 10.

Images