Method and apparatus for categorizing foodstuffs

Abstract

Methods (100) for categorizing (109) foodstuffs, where The procedure (100) comprises the following steps: a) Intake (101) of the food using at least one first sensor (11) of a mobile device (20) b) Evaluation (103) of sensor data from the first sensor (11) and c) Categorization (109) of the foodstuff, characterized by the fact that For the categorization (109) of the food based on sensor data from the first sensor (11) a proportion of an ice layer in the total weight of the food is determined (108).

Description

[0001] The present invention relates to a method and a device for categorizing food using a mobile terminal. State of the art

[0002] Methods and devices for categorizing food are generally known from the state of the art. However, these are typically based on relatively complex technology.

[0003] KR 10 2022 0 089 258 A relates to a system and a method for determining calories from food as well as a method for obtaining depth information and volume information about food.

[0004] US 2021 / 0 405 378 A1 concerns a display system that uses a waveguide.

[0005] The present invention is based on the object of further developing a method and a device for categorizing a food in such a way that food can be categorized reliably and regardless of location.

[0006] The above-mentioned problem is solved by a method for categorizing foodstuffs, which comprises the step of recording the foodstuff using at least one first sensor of a mobile device. Further steps of the method include evaluating sensor data from the first sensor and categorizing the foodstuff.

[0007] The mobile device can be, for example, a smartphone, a tablet, a laptop, glasses or another portable device for perceiving an augmented or virtual reality (augmented reality or virtual reality glasses or wearable).

[0008] A "recording" is understood to mean the acquisition of sensor data from the food. The first sensor acquires sensor data, which is then evaluated and categorized. The food can be, for example, fruit, vegetables, meat, and / or fish. In particular, it can be whole fish, shellfish, crustaceans, or mollusks.

[0009] In particular, at least one image, preferably from different sides, is taken of the outside of the food using the first sensor. The food can be turned between images. In particular, if the food is a fish, shellfish, crustacean, or mollusk, the food can be imaged from at least two sides.

[0010] The term “categorization” of food is understood in particular to mean that the food is categorized according to at least one category, i.e. it is assigned to a respective class of at least one category. For example, the type of food can first be determined. Once the food has been identified, the subspecies can be determined. For example, it can be determined whether the food is fish and, preferably, which type of fish it is. The same applies, for example, to crustaceans, shellfish and mollusks. In the case of meat, too, it can be determined, for example, which type of meat is involved and, in a next step, which cut of meat is involved. One category can therefore be the type of food (e.g. fish or meat) and another category the subspecies of the food (e.g. type of fish).

[0011] The first sensor is preferably designed to scan the food, in other words to measure the entire food using measuring pulses. The first sensor is designed in particular as a laser scanner, especially a lidar sensor. The first sensor is thus designed to emit measuring pulses that are reflected by the food and then received again. The wavelength of the measuring pulse is in particular in the infrared range. From the propagation time of the emitted, reflected and received measuring pulses, depth information of the food can be determined. Furthermore, measuring pulses in the green wavelength range, especially with a wavelength between 522 nm and 542 nm, can be used for lidar measurements. The received measuring pulses in the infrared range can be compared with those in the green wavelength range.

[0012] In particular, the first sensor can be designed as a solid-state lidar sensor. This means that the first sensor does not require movable mirrors or scanning heads to emit measurement pulses at different spatial angles to scan the food. Typically, a corresponding lidar sensor has a transmission matrix and a reception matrix, with the transmission matrix comprising multiple transmission sources. Using transmission optics, measurement pulses from the transmission sources can be emitted in slightly different directions, which are then imaged onto the reception matrix by corresponding reception optics. The transmission elements can be designed primarily as surface emitters, in other words, VCSEL diodes.

[0013] If the first sensor is a lidar sensor, the sensor data is available as a point cloud. For each reflected measurement pulse, a spatial direction—in other words, the angle at which the reflection point is located on the food—as well as distance information are determined. This allows a three-dimensional representation of the food in the form of spatial points, the so-called point cloud, to be created based on all received measurement pulses.

[0014] Because the sensor is part of a mobile device, anyone can take the relevant photos of food. These people don't need to be equipped with special scanning devices; they can take photos of the food themselves, for example, by downloading an application. This allows food to be categorized regardless of location.

[0015] The method may include creating a three-dimensional representation of the food for categorization. In the case of a lidar sensor, the three-dimensional representation is a point cloud, as described above.

[0016] Categorization is understood, in particular, as the assignment to at least one previously defined class of a previously defined category. Examples of a category include, in particular, the type of food, where the class can be defined as fish and / or meat and / or fruit.

[0017] Sensor data from at least one second sensor can also be used to categorize the food. In other words, at least one image, preferably several images, of the food can also be taken using a second sensor, with the sensor data thus obtained being incorporated into the evaluation.

[0018] The first sensor and the second sensor differ in their type and therefore preferably in their measuring principle. The second sensor is, in particular, an infrared sensor, a camera, or a radar sensor.

[0019] The advantage of using a second sensor is that more comprehensive and, if necessary, more precise information about the food can be obtained when two different sensors are used. For example, an infrared sensor can determine the external temperature of the food. A camera, on the other hand, can record the color of the food - compared to a lidar sensor, for example - while a radar sensor can determine depth information. Quantum sensors can be used to determine the chemical composition of the food. For example, contamination of the food, the type of water, or the position of the water in which the food has been located can be determined. Furthermore, physical properties and / or tissue conditions (e.g., whether a food is rotting or whether bacteria are present) or the organic structure can be determined.

[0020] In other words, by using different types of sensors, many characteristics of the food can be determined. A concrete example would be a patterned fish for which both lidar measurement data and image data are available. While the pattern cannot be recognized in the lidar measurement data, it can be determined from the image data. The opposite is true for depth information. This can be derived from the lidar measurement data, but not from the image data. In other words, the pattern is an image-specific feature, while the depth information is a lidar-specific feature. The contour of the food, on the other hand, would be recognizable in both the image data and the lidar data. Using multiple sensors of different sensor types increases the reliability of the categorization.

[0021] In addition to the first sensor, an infrared sensor, a camera, and / or a radar sensor are preferably used. A quantum sensor may also be used.

[0022] Preferably, the sensor data from the first sensor and preferably the sensor data from the second sensor are transmitted, especially wirelessly, to an evaluation unit outside the mobile device for evaluation and categorization. This is therefore an external evaluation unit. The transmission can be carried out, for example, using radio technology, such as mobile communications. Furthermore, the transmission can be carried out via Bluetooth, the Internet, or satellite transmission.

[0023] The evaluation or categorization is thus carried out outside the mobile device. In other words, the evaluation unit is not part of the mobile device. By transmitting the sensor data to an evaluation unit outside the mobile device, the evaluation or categorization is independent of the mobile device, for example of its hardware or software, e.g. software updates. The evaluation unit can in particular be a central processing unit that, for example, receives sensor data from various mobile devices and evaluates sensor data centrally. Therefore, only software- and / or hardware-related requirements for evaluation and categorization must be met with regard to the central evaluation unit. In particular, the evaluation unit can be part of a cloud to which the sensor data is transmitted.Furthermore, the data can be transferred to a hard drive for analysis and categorization. Outsourcing the analysis enables reliable categorization, even if the sensor data is collected from different devices regardless of location.

[0024] The central evaluation unit can have various interfaces, for example, to a third-party processing unit and / or to a manual evaluation module, where categorization is not automated but performed by a person reviewing the sensor data. Furthermore, the evaluation unit can include a module for automated evaluation, especially using artificial intelligence.

[0025] The analysis primarily involves extracting characteristics of the food from the sensor data and comparing these characteristics with stored data of previously defined classes. Based on the comparison, the food is assigned to a class. When analyzing image data, for example, the images can be compared with previously stored image data.

[0026] Above all, the evaluation and / or categorization is automated and therefore not based on a person viewing the sensor data. The evaluation unit can primarily use artificial intelligence for evaluation and / or categorization.

[0027] The evaluation unit can, in particular, comprise one feature extractor unit for each sensor type of the corresponding sensor data. For example, the evaluation unit can comprise one feature extractor unit for sensor data from a lidar sensor and another feature extractor unit for image data, i.e., sensor data from a camera. The feature extractor units serve to extract features from the measurement data.

[0028] Each feature extraction unit can comprise at least one neural network for evaluation. In particular, the feature extraction units are neural networks. For example, the feature extraction unit for image data can have a convolutional neural (CNN) network that comprises at least one convolutional layer. A convolutional neural network is capable of processing input in the form of an image. For example, a CNN network can extract simple features such as straight lines in the first layers of the network and more complex features such as L-features, circle features, etc. in deeper layers. A corresponding feature extraction unit for lidar data, i.e., sensor data from a lidar sensor, can be designed, for example, as a multilayer perceptron (MLP) network, wherein the MLP network can have a plurality of MLP layers.The MLP network is capable of processing input in the form of point clouds.

[0029] Furthermore, the evaluation unit can comprise a feature comparison unit that can compare correspondingly extracted features with stored data of previously defined classes. In particular, both the feature extraction unit and the feature comparison unit are trained using unsupervised learning. In other words, the corresponding neural networks comprise weights determined by corresponding unsupervised learning. Training involves providing an input, wherein the output of the corresponding network is compared with the input in order to determine the corresponding weights of the network.

[0030] In particular, categorization occurs by comparing extracted features with stored data from previously defined classes of at least one category. For example, by training the corresponding neural networks, an average vector can be assigned to each class of each category. Features extracted from lidar data, which are typically present as vectors, can be compared with the corresponding average vectors of the corresponding classes, for example, by determining the cosine similarity. The feature can then be assigned, for example, to the class with the greatest cosine similarity, thus enabling categorization.

[0031] For example, different fish species can be stored. These are characterized in particular by their three-dimensional shape. This allows certain three-dimensional shapes to be assigned to a specific fish species. The three-dimensional shape of a fish can be deduced from corresponding image or lidar data, whereby the three-dimensional shape can be compared with the stored three-dimensional shapes of previously defined classes. An assignment can be made based on the greatest similarity across all classes.

[0032] The method can also be used to assign a confidence level to the categorization, especially for each category. This confidence level can be displayed to the user, providing transparency about the degree of certainty with which the food in question has been categorized.

[0033] To categorize the food, a color and / or a temperature, in particular an outside temperature, of the food can be determined, for example, based on the sensor data from the first sensor and / or the second sensor. For example, the color can be determined using sensor data from a camera, while the temperature is preferably determined using sensor data from an infrared camera. The color and / or the temperature can therefore be respective characteristics of the food that are used for categorization. For example, the temperature can be used to determine whether the food is frozen. A category can therefore be the state, e.g. frozen or not frozen, with the temperature being extracted as a characteristic and compared with previously stored temperature data. Furthermore, conclusions can be drawn about the hardness of the food based on the temperature.In particular, frozen foods are typically harder than non-frozen foods. For example, different degrees of hardness can be defined in the hardness category, with frozen foods being assigned to a higher hardness class.

[0034] To categorize the food, a three-dimensional shape and / or a surface, especially a surface texture, can be determined based on the sensor data. A three-dimensional shape of the food can be determined, for example, from sensor data from a camera and / or a lidar sensor. Based on the three-dimensional shape, the food type and / or food subtype, e.g., fish species, can be determined. For this purpose, the shape is compared with previously stored shapes related to the individual categories or their classes.

[0035] The surface is determined primarily based on sensor data from a lidar sensor, as this contains the necessary depth information. Based on this depth information, it can be determined whether the surface is smooth, for example, or whether it has significant unevenness.

[0036] Conclusions about the quality of the food can be drawn based on the surface. In particular, the smoothness of the surface is determined.

[0037] Some foods are characterized by a very smooth surface, so any unevenness on this surface indicates degraded quality. Therefore, there may be different classes within the quality category that allow a classification of quality based on the surface.

[0038] Furthermore, based on the surface, especially for certain foods, conclusions can be drawn about additives or ingredients in the food, since it is known, for example, that these additives or ingredients in particular are responsible for corresponding surface damage. One example would be phosphate. A degraded surface texture and thus also a degraded quality is typically caused by phosphate in some foods, such as fish, so detecting a degradation in smoothness can indirectly determine the presence of phosphate.

[0039] It can also be determined whether the food is glazed. Glazed means that the food is at least partially, preferably completely, covered with a layer of ice (the glaze). This can be done primarily using the sensor data based on the lidar scanner. To categorize the food, according to the invention, a proportion (preferably as a percentage) of an ice layer in relation to the total weight of the food is determined on the basis of the sensor data from the first sensor. In particular, a weight of the food is estimated on the basis of the sensor data. There can therefore be various gradations in the weight category, whereby a weight can be estimated, for example, based on the detected type of food, preferably the subtype of food, and the three-dimensional shape. For this purpose, a length and / or width and / or a diameter and / or a volume of the food can be estimated primarily from the shape.This allows an estimated total weight of the food to be determined and preferably assigned to a calibration. Especially for fish products, crustaceans, shellfish, and mollusks, this allows for assignment to standard calibrations, specifying the weight including or excluding the glaze content.

[0040] Using a lidar sensor, the thickness of an ice layer on the frozen food can also be determined. For example, it can be determined whether only a minimal protective glaze is present (and thus a glaze that does not have to be declared according to the Food Information Regulation (LMIV)). Measurement pulses from a corresponding lidar measuring method are reflected both by the outer ice layer and by the transition between the ice layer and the food, so that the thickness of the ice layer can be determined based on the sensor data from a lidar sensor. The thickness of the ice layer, in turn, can be used to determine the proportion of the ice layer in the total weight. Such information is particularly important for fish, shellfish, crustaceans, and mollusks, as these are often frozen and their prices are often based on their weight, whereby a maximum ice content must not be exceeded.The presence of this maximum ice fraction can thus be easily monitored using this method. In particular, the lidar sensor uses infrared and / or green light. In particular, measurement pulses in the green wavelength range, especially with a wavelength between 522 nm and 542 nm, can be used for lidar measurements. The received measurement pulses in the infrared range can be compared with those in the green wavelength range. Based on the green measurement pulses, in particular, the transition between the fish's outer skin and the ice can be determined very precisely.

[0041] Furthermore, the origin of the food can be determined based on the sensor data. For example, in the case of a fish, it can be determined which type of water (freshwater or saltwater) and / or which location, e.g., which sea, it comes from. This can then be used to verify the origin, for example. The sensor data from the quantum sensor can be used for this purpose. The quantum sensor can be used to determine the chemical, particularly mineral, composition of the residual water still present in the food. The determined composition can be compared with previously stored compositions from specific locations, e.g., seas, and thus a location with the greatest similarity can be identified. The type of water (freshwater or saltwater) and / or the location can be compared with the food subspecies, e.g., fish species, which can increase the certainty about the food's place of origin.This helps prevent fraud and monitor compliance with standards, such as sustainability.

[0042] If the food is a liquid, the gas composition can also be determined using the sensor data. The gas composition can also be determined based on sensor data from a quantum sensor.

[0043] It can also be determined whether the food is still alive. Based primarily on the green light sensor data, blood flow can be determined in close range to determine whether an object is alive, or its blood viscosity can be determined.

[0044] The method may involve taking multiple images of the food over a certain period of time to monitor its development. For example, multiple images can be taken, based on which a video sequence can be created. This allows the temporal development of characteristics or categorization results to be observed over time.

[0045] The results of the categorization can be stored, in particular, using a storage unit. Furthermore, the sensor data and / or previously determined characteristics and / or the results of the categorization can be transmitted to third parties, e.g., suppliers, transport companies, customers, etc. Furthermore, the data can be processed via at least one payment system, an e-commerce platform, a logistics platform, a distributed ledger technology platform, or a blockchain setup to enable complete transparency throughout the entire supply chain. The corresponding three-dimensional representation can also be passed on accordingly. For example, on an e-commerce marketplace, the food can thus be presented three-dimensionally, in particular together with the determined categories. The categorization can be used, for example, to identify deficiencies in compliance with supply chain requirements.This can result, for example, from a deviation from the identified class regarding the quality category. By sharing the results of the categorization with third parties, the deficiency in the supply chain can be easily identified and remedied due to maximum transparency.

[0046] Third parties may use the provided data, in particular, for calculations and processing purposes for payment. Furthermore, the information may be used to facilitate transport, for example, by forwarding it to logistics companies in advance. In this context, the three-dimensional shape, estimated weight, and dimensions are particularly relevant.

[0047] Furthermore, the corresponding sensor data or results of the evaluation and categorization can be used to verify the maintenance of international quality with regard to production standards, for example for local fisheries or meat producers.

[0048] The corresponding recorded sensor data or the results of the evaluation or categorization can each be assigned to a user profile, for example, within an Apple ID or a Google Play account. In particular, the method can include adapting the available categories or classes for certain categories to a respective user. If, for example, a user profile is used exclusively for fish, other classes of the food category can be excluded. This facilitates categorization. This dynamic adaptation can be carried out, for example, based on previously recorded sensor data or results of evaluation and categorization.

[0049] Sensor data and / or characteristics and / or results of the food's categorization can be transformed into virtual or augmented reality, thus visually representing the food in virtual or augmented reality. Furthermore, detected food characteristics can be used to update a virtual inventory list.

[0050] Another application of this method is fish detection during fishing. Based on this method, a fisherman can identify characteristics of caught fish, for example, to check whether they are legally permissible for capture. This allows a fisherman to determine whether fish are, for example, too small or endangered and release them back into the wild.

[0051] In a further aspect, the present invention relates to a device which comprises at least a first sensor of a mobile terminal for receiving a food item and at least one evaluation unit for evaluating sensor data from the first sensor and for categorizing the food item. According to the invention, the device is designed to determine a proportion of an ice layer in relation to the total weight of the food item in order to categorize the food item on the basis of sensor data from the first sensor. In this case, the evaluation unit is in particular not part of the mobile terminal. The device can further comprise a transmission unit of the mobile terminal for transmitting sensor data to the external evaluation unit. The device is designed in particular to carry out the method described above. For this purpose, the device can in particular comprise at least one second sensor. Short description of the characters

[0052] They show in a purely schematic representation: Fig. 1 a flow diagram of a method according to the invention and Fig. 2 a device according to the invention. Detailed description of the characters

[0053] Fig. Figure 1 shows a process diagram of a method 100 according to the invention for categorizing foodstuffs. The method 100 comprises the recording 101 of the foodstuff using at least one first sensor of a mobile terminal. The method 100 further comprises the evaluation 103 of the sensor data and the categorization 109 of the foodstuff.

[0054] In addition to the sensor data of the first sensor, sensor data from at least one second sensor can also be used for evaluation 104. In such a case, the food is also recorded using the second sensor.

[0055] The evaluation unit is, in particular, an external evaluation unit, so that the sensor data is transmitted to the evaluation unit 102. A three-dimensional shape of the food is created 105.

[0056] During the evaluation and to categorize the food, a color and / or a temperature of the food is determined 106 and / or a three-dimensional shape and / or a surface is determined 107 and / or a proportion of an ice layer in the total weight of the food is determined 108. All of the aforementioned features serve to categorize 109 the food. The method 100 can include taking 110 multiple images of the food over a certain period of time in order to observe the aforementioned features over time.

[0057] The sensor data and / or previously determined characteristics of the food can be transmitted to a third party 111. This occurs, in particular, via an interface. Characteristics of the food can be transformed into a virtual or augmented reality 112, thus allowing the food to be visually represented in virtual or augmented reality 113. Furthermore, determined characteristics of the food can be used to update 114 a virtual inventory list.

[0058] In Fig.Figure 2 shows a device 10 according to the invention for categorizing a food item 30. The device 10 comprises a first sensor 11 of a mobile terminal 20, wherein the first sensor 11 is configured to capture images of the food item 30, which in this case is a fish. In particular, the first sensor 11 can scan the food item. Furthermore, the device 10 has a second sensor 12, which is also configured to capture the food item.

[0059] The acquired sensor data can be sent to an external evaluation unit 14 via a transmission unit 13 of the mobile terminal 20. The external evaluation unit 14 is, in particular, a central processing unit. This, in turn, can have various interfaces 15, for example, to a third-party processing unit. For example, the sensor data can be made available to a person using an interface 15 so that the person can manually categorize the food. Beforehand, the evaluation unit 14 can, in particular, display the food in a three-dimensional form. List of reference symbols 100 procedures 101 Recording of the food by means of at least one first sensor of a mobile device 102 Transmission of sensor data to an evaluation unit outside the mobile device 103 Evaluation of the sensor data of the first sensor 104 Use of sensor data from at least one second sensor 105 Creating a three-dimensional representation 106 Determining the color and / or temperature of the food 107 Determining a three-dimensional shape and / or a surface 108 Determining the proportion of an ice layer to the total weight of the food 109 Categorization of the food 110 Taking multiple photographs of the food over a period of time 111 Transfer of sensor data and / or determined characteristics of the food to a third party 112 Transforming food characteristics in a virtual or augmented reality 113 Visual representation of the food in a virtual or augmented reality 114 Updating a virtual inventory list 10 Device 11 first sensor 12 second sensor 13 Transmission unit 14 Evaluation unit 15 Interface 20 mobile devices 30 groceries

Claims

[1] Procedure (100) for categorising (109) foodstuffs, where the method (100) comprises the following steps: a) Recording (101) of the food by means of at least one first sensor (11) of a mobile terminal (20) b) evaluation (103) of sensor data of the first sensor (11) and c) categorisation (109) of the food, characterized by , that to categorize (109) the food, a proportion of an ice layer in the total weight of the food is determined (108) on the basis of sensor data from the first sensor (11). [2] Method (100) according to claim 1, characterized by that the first sensor (11) is designed as a laser scanner, in particular a lidar scanner. [3] Method (100) according to claim 1 or 2, characterized by that the method (100) comprises creating (105) a three-dimensional representation of the food for categorization (109). [4] Method (100) according to one of the preceding claims, characterized by that for categorising (109) the food, sensor data from at least one second sensor (12), namely an infrared sensor or a camera or a radar sensor, are further used (104). [5] Method (100) according to claim 4, characterized by that sensor data of the first sensor (11) and preferably sensor data of the second sensor (12) are transmitted (102) to an evaluation unit (14) outside the mobile terminal (20) for evaluation and categorization. [6] Method (100) according to one of claims 4 or 5, characterized by that for the categorization (109) of the food, a color and / or a temperature of the food is determined (106) on the basis of sensor data from the first sensor (11) and / or the second sensor (12). [7] Method (100) according to one of claims 4 to 6, characterized bythat for the categorization (109) of the food, a three-dimensional shape and / or a surface is determined (107) on the basis of sensor data from the first sensor (11) and / or the second sensor (12). [8] Method (100) according to one of claims 4 to 7, characterized by that for the categorisation (109) of the food, a proportion of an ice layer in the total weight of the food is additionally determined (108) on the basis of sensor data from the second sensor (12). [9] Method (100) according to one of the preceding claims, characterized by that the method (100) may comprise performing (110) multiple images of the food over a certain period of time in order to monitor a development of the food. [10] Device (10) for categorising (109) foodstuffs, wherein the device (10) comprises a first sensor (11) of a mobile terminal (20) and an evaluation unit (14) for evaluating (103) sensor data of the first sensor (11) and for categorizing (109) the food, characterized by , that the device (10) is designed to determine (108) a proportion of an ice layer in the total weight of the food for the purpose of categorizing (109) the food on the basis of sensor data from the first sensor (11).

Images