System and method for acquiring a plurality of images and associated identification system and method

Abstract

The invention relates to an image acquisition system (10) of a food product (5, 6) arranged in an acquisition zone (1), the acquisition system comprising: - a camera (12) configured to acquire a color image, the camera having a first optical axis, - a thermal detector (14) configured to acquire a thermal image, the thermal detector having a second optical axis, - a three-dimensional depth sensor (16) configured to acquire a relief image, the three-dimensional depth sensor having a third optical axis, - a processing unit (15) configured to receive each image and to generate three final images, the thermal detector being positioned between the camera and the three-dimensional depth sensor, the first optical axis, the second optical axis and the third optical axis being parallel to each other,the first optical axis and the third optical axis being spaced between 5 and 20 centimeters apart. Figure for the abbreviation: Fig. 2,

Description

Title of the invention: System and method for acquiring a plurality of images and associated identification system and method Technical field of the invention

[0001] The present invention relates generally to the field of product identification, in particular food products.

[0002] It finds a particularly advantageous application in the field of collective catering, to identify products positioned on a tray, in particular with a view to their automatic invoicing. It also finds a privileged application in the estimation of the nutritional and calorific characteristics of foods positioned on a tray. Finally, it finds an advantageous application in the context of the quantification of food waste.

[0003] It relates more particularly to a system for acquiring images of a food product placed in an acquisition zone. State of the art

[0004] In order to speed up the flow of people in collective catering establishments, it is currently being sought to carry out faster cash collections.

[0005] Document FR3069685 discloses in particular an installation aimed at increasing the speed of collection to facilitate and smooth the movement of people in the product distribution area.

[0006] According to this document, this installation comprises an acquisition station equipped with at least one camera oriented towards a surface for placing a tray comprising the different products selected by a user. According to this document, the acquisition station comprises three cameras making it possible to acquire images of the products present on the tray from different angles. The acquired images are then processed to automatically identify the different products positioned on the tray and then issue the corresponding payment receipt.

[0007] Such an installation effectively makes it possible to automate the issuance of a ticket and therefore to reduce human intervention to generate the collection ticket. However, the identification of products by means of this installation has limited reliability, in particular for example when a product is partially or totally hidden by a plate or a paper napkin or when it is a question of identifying different products, of different prices and nutritional values, but of identical colors, containers or other visual aspects. Presentation of the invention

[0008] The present invention proposes to improve the automatic identification of products food placed on a tray, particularly when they are likely to be partially hidden by another product or a napkin or in the case of food products at different temperatures.

[0009] To this end, the invention relates to a system for acquiring a plurality of images of at least one food product placed in an acquisition zone. According to the invention, the acquisition system comprises: - a camera configured to acquire at least one color image of the acquisition area, the camera having a first optical axis, - a thermal detector configured to acquire at least one thermal image of the acquisition area, the thermal detector having a second optical axis, - a three-dimensional depth sensor configured to acquire at least one image of the relief of the acquisition area, the three-dimensional depth sensor having a third optical axis, and - a processing unit configured to receive each image and to generate three final images of the acquisition area, the thermal detector being positioned between the camera and the three-dimensional depth sensor, the first, second and third optical axes being parallel to each other, the first optical axis and the third optical axis being spaced apart by a distance of between 5 and 20 centimeters.

[0010] The invention thus allows the acquisition of different types of images of the acquisition zone and therefore of the food product. In particular, the acquired images are not limited only to an acquisition of color images of the acquisition zone but also to an acquisition of thermal data and depth data. This then makes it possible to characterize the food product more precisely, by having visual characteristics of the food product but also other physical characteristics such as its temperature, its dimensions and its volume.

[0011] Furthermore, the relative arrangement of the different elements of the acquisition system makes it possible to ensure the quality of the images, in particular by avoiding the representation of the acquisition zone on the different images being truncated.

[0012] The images obtained are therefore particularly advantageous for subsequently enabling precise identification of the food product.

[0013] According to a variant of the invention, the first optical axis and the third optical axis are preferably spaced 6 centimeters apart.

[0014] This relative arrangement of the camera, the thermal detector and the three-dimensional depth sensor makes it possible to ensure the quality of the images, in particular by allowing a complete representation of the acquisition zone. All the food products present in the acquisition zone are therefore visible by the camera, thermal detector and three-dimensional depth sensor.

[0015] According to one embodiment of the invention, the first optical axis, the second optical axis and the third optical axis are contained in the same plane.

[0016] This makes it possible to benefit from possible parallax effects which make it possible to enrich the visual information obtained from the images acquired by several cameras making it possible to acquire color images, in terms of depth and perspective.

[0017] According to another embodiment of the invention, the first optical axis and the third optical axis being contained in a first plane, the second optical axis is contained in a second plane parallel and distinct from the first plane. In other words, the projections of the three optical axes in a plane normal to the latter are not aligned.

[0018] According to the invention, the distance between the first plane and the second plane is less than or equal to 5 centimeters.

[0019] This allows a more compact arrangement of the camera, the thermal detector and the three-dimensional depth sensor while ensuring satisfactory quality of the images obtained.

[0020] According to a characteristic of the invention, the processing unit is configured to obtain two distinct color images of the acquisition zone according to respectively a first resolution and a second resolution and four final images are formed of an image in each of the two resolutions, of the thermal image and of the image of the relief of the acquisition zone. The first resolution is higher than the second resolution. According to a variant of the invention, the first resolution corresponds to a high resolution and the second resolution corresponds to a medium resolution.

[0021] According to a characteristic of the invention, the processing unit is configured to obtain two distinct color images of the acquisition zone, each obtained respectively according to a first viewing angle and according to a second viewing angle, and the four final images are formed from two color images, each according to a viewing angle, of the thermal image and of the relief image of the acquisition zone.

[0022] The use of these four different types of images allows for more precise identification of the food products present in the acquisition area. In addition, the acquisition of the two color images with two different resolutions is particularly easy to implement.

[0023] According to another variant of the invention, the processing unit is configured to obtain three distinct color images of the acquisition zone, each obtained respectively according to a first exposure, a second exposure and a third exposure, five final images are therefore formed from a color image in each of the first exposure, the second exposure and the third exposure, the thermal image and the image of the relief of the acquisition zone.

[0024] The use of images with different exposures, an underexposed image and an overexposed image, makes it possible to limit the loss of information concerning the dark parts (often poorly lit in practice) or the very light parts (often subject to strong lighting) of the acquisition zone.

[0025] According to another characteristic of the invention, the color image being decomposed into three primary colors, the five final images are formed from an image in each of the three primary colors, the thermal image and the image of the relief of the acquisition zone. Preferably, the decomposition is a decomposition according to the colors red green blue or RGB for Red, Green, Blue in English.

[0026] The use of these five different types of images allows for more precise identification of the food products present in the acquisition area.

[0027] According to another variant of the invention, the acquisition system also comprises a device for lighting the acquisition area configured to emit continuous illumination or instantaneous illumination of the acquisition area. Preferably, the lighting device emits white light whose color temperature is between 3500K and 6400K.

[0028] Such a lighting device makes it possible to correct the quality of the ambient lighting of the place where the device according to the invention is located. This lighting device is used in addition to the natural light present in the environment of the acquisition zone. This makes it possible to improve the quality of the images obtained by preventing certain parts of the acquisition zone from being invisible or barely visible and / or by correcting the color rendering.

[0029] According to yet another variant of the invention, the camera comprises a polarizing filter.

[0030] This polarizing filter aims to attenuate, or even cancel, possible reflections on the specular parts of objects and products located in the acquisition zone.

[0031] According to an alternative embodiment of the invention, the acquisition system also comprises communication means connected to the processing unit and configured to exchange data with a remote server.

[0032] The invention also relates to a system for identifying at least one food product, the identification system comprising:

[0033] - an acquisition zone adapted to receive the food product,

[0034] - a system for acquiring, according to the invention, a plurality of images of each food product, and

[0035] - a processing system configured to receive the three images as input generated by the processing unit and to provide output identification data for each food product present in the acquisition area.

[0036] This identification system makes it possible, from the acquired images, to identify the food products present in the acquisition area. The identification data can then be used for billing or inventory applications.

[0037] According to a variant of the invention, the identification system also comprises a remote server configured to receive the three images generated by the processing unit of the acquisition system and the identification data determined by the identification system processing system and to transmit the identification data to an external entity.

[0038] Thus, the determined identification data can be transmitted to external entities, such as for example a cash register device in the context of collective catering to enable the food products chosen by the user concerned to be invoiced.

[0039] According to another variant of the invention, the acquisition system is positioned at a distance of between 50 and 90 centimeters from the acquisition zone and, preferably, between 50 and 65 centimeters from the acquisition zone, more preferably, between 55 and 60 centimeters from the acquisition zone.

[0040] It has been found that such distance ranges make it possible to obtain an acquisition range with an optimal depth of field for good recognition of the products while preserving ease of access compatible with the size of the trays and the products likely to be found there.

[0041] The invention also relates to a method for acquiring a plurality of images of at least one food product placed in an acquisition zone, the method comprising steps of: - acquisition of at least one color image of the acquisition area, - acquisition of a thermal image of the acquisition area, - acquisition of a relief image of the acquisition area, and - generation of three final images from each image, the three final images being formed from a color image, the thermal image and the relief image of the acquisition area.

[0042] Within the meaning of the invention, a color image can be formed either from a monochrome image in a primary or component color or from an image formed from a variation of gray levels or from a polychrome image integrating three primary colors.

[0043] The invention finally relates to a method for identifying at least one food product placed in an acquisition zone, the identification method comprising steps of: - acquisition of a plurality of images of each food product according to the acquisition method in accordance with the invention, and - identification of each food product by providing data identification of each food product present in the acquisition area.

[0044] Of course, the various features, variants and embodiments of the invention may be combined with each other in various combinations to the extent that they are not incompatible or mutually exclusive. Detailed description of the invention

[0045] The description which follows with reference to the appended drawings, given as non-limiting examples, will make it clear what the invention consists of and how it can be implemented.

[0046] [Fig-1] represents a schematic view from above of a conformal acquisition zone to the invention,

[0047] [Fig.2] schematically represents an image acquisition system according to the invention,

[0048] [Fig.3] represents a diagram illustrating the positioning of a camera, a thermal detector and a three-dimensional depth sensor in the image acquisition system of [Fig.2],

[0049] [Fig.4] represents an example of a thermal image of the acquisition zone, obtained within the framework of the present invention,

[0050] [Fig.5] represents an example of an image of the relief of the acquisition zone, obtained within the framework of the present invention,

[0051] [Fig.6] schematically represents an identification system in accordance with the invention,

[0052] [Fig.7] represents, in the form of a flowchart, an example of an image acquisition method in accordance with the invention, and

[0053] [Fig.8] represents, in the form of a flowchart, an example of an identification method in accordance with the invention.

[0054] The invention aims, according to a first application, to enable the automatic identification of products or foods in order to facilitate or even automate the process of invoicing the price of these products, particularly in a collective catering context. To this end, the invention proposes, as shown schematically in [Fig. 1], to implement an identification system S which communicates via a communication network R with a payment or invoicing terminal T.

[0055] The identification system S then comprises an acquisition zone 1 intended to receive products or food to be identified as will appear later. The acquisition zone 1 is associated with an image acquisition system 10 which, according to the example illustrated, is placed above it. The identification system communicates via a communication network 2 with an identification server SRV in order in particular to allow the establishment of an invoice for the products 5, 6 in located in acquisition zone 1. According to the example illustrated, the products 5, 6 to be identified are placed on a serving tray of the type used in collective catering by customers to transport the items chosen for their meal. The tray is then placed in acquisition zone 1 by a customer wishing to have the collection ticket corresponding to his selection established.

[0056] Thus, the invention relates in particular to an image acquisition system 10 as illustrated in [Fig. 2] and designated as a whole by the reference 10. According to this exemplary embodiment, the image acquisition system 10 comprises a camera 12, a thermal detector 14, a three-dimensional depth sensor 16, a processing unit 15 and communication means 18.

[0057] Advantageously according to the invention, the image acquisition system 10 is positioned at a distance h from the acquisition zone 1. This distance h is here between 50 and 90 cm. Preferably, this distance h is between 50 and 65 cm. More preferably, this distance is between 55 and 60 cm. This arrangement of the image acquisition system 10 relative to the acquisition zone 1 makes it possible to capture the entire acquisition zone 1 and to obtain images (whatever their type) of good quality to allow analysis by a processing system 24 of the identification system S.

[0058] As shown in [Fig. 3], the camera 12, the thermal detector 14 and the three-dimensional depth sensor 16 are positioned on a support 19. The support 19 has a generally rectangular shape. The front face 19A of the support 19 materializes a plane P.

[0059] As shown in [Fig.3], the thermal detector 14 is here positioned between the camera 12 and the three-dimensional depth sensor 16.

[0060] Each of the camera 12, the thermal detector 14 and the three-dimensional depth sensor 16 are positioned on the support 19 so that their respective optical axes are perpendicular to the support 19. By definition, the optical axis of an optical system is defined as the axis of rotational symmetry of this optical system.

[0061] Thus, as shown in [Fig. 3], the camera 12 has a first optical axis A1, orthogonal to the plane P of the support 19. The thermal detector 14 has a second optical axis A2, orthogonal to the plane P of the support 19. The three-dimensional depth sensor 16 has a third optical axis A3, orthogonal to the plane P of the support 19. The first optical axis A1, the second optical axis A2 and the third optical axis A3 are parallel to each other.

[0062] Advantageously according to the invention, the first optical axis A1 and the third optical axis A3 are spaced apart by a distance d of between 5 and 20 centimeters (cm). This relative arrangement of the camera 12, the thermal detector 14 and the three-dimensional depth sensor 16 makes it possible to ensure the quality of the images, in allowing in particular a complete representation of the acquisition zone 1. All the food products 5, 6 present in the acquisition zone 1 are therefore visible by the camera 12, the thermal detector 14 and the three-dimensional depth sensor 16.

[0063] Preferably, the first optical axis A1 and the third optical axis A3 are spaced apart by a distance d of the order of 6 cm.

[0064] In the example of [Fig. 3], the first optical axis A1 and the third optical axis A3 are contained in a first plane PI. The second optical axis A2 is, for its part, contained in a second plane P2, parallel and distinct from the first plane PI. In other words, this means that, for their positioning on the support 19, the camera 12, the thermal detector 14 and the three-dimensional depth sensor 16 are not aligned (here only the camera 12 and the three-dimensional depth sensor 16 are aligned according to the same plane PI).

[0065] Advantageously, the first plane PI and the second plane P2 are spaced apart by a distance di of less than 5 cm. This allows a more compact arrangement of the camera 12, the thermal detector 14 and the three-dimensional depth sensor 16 while guaranteeing satisfactory quality of the images obtained.

[0066] The camera 12 is here an image acquisition system comprising a photographic sensor of the CMOS sensor type (for “Complementary Metal-Oxide-S emi-conductor”). Such a sensor is notably characterized by, on the one hand, the dimensions of its rectangular photosensitive surface which can be summarized by its format and the length of the diagonal thereof, and, on the other hand, its resolution.

[0067] The sensor implemented has for example a format of type 1 / 2.3' with a diagonal greater than 7 mm and a resolution between 2028x1520 and 8112x6080. In the present case, a sensor is implemented having an image format of 1 / 2.3, a diagonal of the order of 7.9 mm and a resolution of 4056 x 3040 such as the IMX 477 sensor marketed by Sony. Such a sensor has the advantage of having good sensitivity so that it is possible to obtain good quality images in ambient light without additional lighting. Thus, this makes it possible to avoid parasitic reflections induced by additional lighting and which harm the quality of the acquired images. Furthermore, such a sensor represents a good compromise between performance and the associated cost.

[0068] The photographic sensor is associated with an optic making it possible to acquire an image encompassing the entire acquisition zone. This optic has, for example, a fixed focal length combined with the use of a diaphragm which is also fixed.

[0069] Optionally, this optic, and therefore the camera, can be equipped with a polarizing filter. This polarizing filter aims to attenuate, or even cancel, the effects of possible reflections due to the nature of the products present in the acquisition zone 1. By For example, plastic packaging around food products 5, 6 can be the cause of such reflections.

[0070] The camera 12 makes it possible to acquire color images of the acquisition zone 1 as will appear later.

[0071] The image acquisition system 10 also comprises the thermal detector 14.

[0072] This thermal detector 14 is for example an infrared camera comprising an infrared sensor associated with an optic making it possible to produce a thermal image encompassing at least the entire acquisition zone. This is mid-infrared, therefore wavelengths between 3 and 50 micrometers (pm). In the present case, the field of vision of this thermal detector 14 is for example 55 degrees by 35 degrees or 110 degrees by 75 degrees. The resolution of the thermal detector 14 is here at least 16x12 pixels. It is for example between 16x12 pixels and 64x48 pixels and, preferably, 32x24 pixels.

[0073] Here, the thermal detector 14 is for example the Melexis MLX90640 model.

[0074] The thermal detector 14 is configured to acquire at least one thermal image Ith of acquisition zone 1 as will be described later.

[0075] The image acquisition system 10 also includes the three-dimensional depth sensor 16.

[0076] This three-dimensional depth sensor 16 comprises a stereoscopic sensor. The field of vision of the three-dimensional depth sensor 16 is for example 65 degrees by 40 degrees. The resolution of this sensor is here between 640x360 pixels and 2560x1440, preferably 1280x720 pixels.

[0077] Here, the three-dimensional depth sensor 16 is for example a depth camera of the Intel® RealSense™ D415 type.

[0078] The three-dimensional depth sensor 16 is configured to acquire at least one image of the relief Idpth of the acquisition zone 1, as described later.

[0079] The three-dimensional depth sensor 16 also comprises a control unit and an image acquisition sensor. The three-dimensional depth sensor 16 is then also configured to acquire a color image of the acquisition area 1.

[0080] As shown in [Fig.2], the image acquisition system 10 also comprises a lighting device 11. This lighting device 11 is used in addition to the ambient light present in the environment of the acquisition zone 1. This lighting device can, for example, make it possible to obtain diffusing lighting in the environment of the acquisition zone 1.

[0081] This lighting device 11 is for example configured to emit illumination continuous from acquisition zone 1. This is, for example, a lamp present in the environment of the acquisition zone.

[0082] As shown in [Fig.2], the image acquisition system 10 also comprises the processing unit 15. This processing unit 15 comprises, for example, a processor and a memory and ensures the control of the camera 12, the thermal detector 14 and the three-dimensional depth sensor 16 as well as the processing of the images resulting from these.

[0083] As described below, the processing unit 15 is configured to trigger the acquisition of the different images by the camera 12, the thermal detector 14 and the three-dimensional depth sensor 16. The processing unit is also configured to receive each image acquired by the camera 12, the thermal detector 14 and the three-dimensional depth sensor 16 and to generate at least three final images of the acquisition zone 1 (an RGB color image, a thermal image Ith and a relief image Idpth). These three final images can then be analyzed so as, for example, to identify the food products 5, 6 located in the acquisition zone 1.

[0084] The processing unit 15 makes it possible in particular to ensure the correct alignment of the images, to adjust the perspectives associated with each acquired image and to resize them so that each image is centered on the acquisition zone. It also makes it possible to modify the format of the acquired images so that they can be analyzed subsequently. In particular, the processing unit 15 receives, for example, in practice, a raw file from each camera and sensor associating, with each pixel, a piece of data. The processing unit 15 is then configured to transform this data so that it can be used for the identification of the food products 5, 6. The processing unit 15 can also compress the images so that they can be stored and transmitted more conveniently.The processing unit 15 can also implement a step of anonymizing the images received in order to ensure the confidentiality of the consumer's data, such as for example data relating to bank cards, mobile phones or any other non-food item possibly placed on the tray.

[0085] In a preferred embodiment, the processing unit is configured to obtain two distinct RGB color images of the acquisition zone 1 according to a first resolution and a second resolution respectively. Four final images are then formed here by a first RGB color image having a first resolution, a second RGB color image having a second resolution, a thermal image Ith and a relief image Idpth of the acquisition zone 1. The first resolution is here greater than the second resolution. Here, the first resolution is said high and is for example 2560x1440 pixels or 2560x1600 pixels or 3840x2160 pixels. The second resolution is called medium and is for example 1920x1080 pixels or 1920x1200 pixels.

[0086] Finally, the image acquisition system 10 also comprises the communication means 18. These communication means 18 are, for example, wireless communication means making it possible to establish wireless communication with a remote entity. These communication means 18 are, for example, configured to exchange data with a remote server 25. Here, the communication means 18 are, for example, configured to exchange the three (or more depending on the embodiment) final images generated with a remote server, making it possible, for example, to store them externally to the image acquisition system 10 (for example, for long-term backup of these images).

[0087] Certain variant embodiments of the image acquisition system 10 can be implemented.

[0088] According to an alternative embodiment, the first optical axis, the second optical axis and the third optical axis can be contained in the same plane.

[0089] According to another embodiment variant, the lighting device can be omitted from the image acquisition system. Indeed, it is not strictly necessary, in particular for environments in which the ambient lighting is sufficient.

[0090] According to another alternative embodiment, the lighting device can be configured to emit instantaneous illumination of the acquisition zone.

[0091] According to yet another variant embodiment, the image acquisition system can make it possible to obtain four final images formed from two distinct color images of the acquisition zone, each obtained respectively according to a first viewing angle and according to a second viewing angle, from the thermal image and from the relief image of the acquisition zone.

[0092] According to yet another variant embodiment, the image acquisition system can make it possible to obtain five final images formed by three color images, each in a primary color, a thermal image and an image of the relief of the acquisition zone.

[0093] According to yet another embodiment variant, the image acquisition system can make it possible to obtain five final images formed by an RGB color image, an underexposed image, an overexposed image, a thermal image and an image of the relief of the acquisition zone. In this variant, the overexposed image is obtained using a long exposure time and the underexposed image is obtained using a short exposure time. The RGB color image then corresponds to an average exposure compared to the other two images (underexposed and overexposed). The use of underexposed and overexposed images makes it possible to limit the loss information concerning the dark parts (often poorly lit in practice) or the very light parts (often subject to strong lighting) of the acquisition area.

[0094] The present invention aims to identify the food products 5, 6 present in the acquisition zone 1, from the images acquired by the acquisition system 10.

[0095] [Fig.6] schematically represents an identification system S of at least one food product 5, 6 present in the acquisition zone 1.

[0096] As shown in [Fig. 6], this identification system S comprises the image acquisition system 10 described previously. This image acquisition system 10 therefore makes it possible to acquire images (for example three, four or five images according to the embodiments introduced previously) of the acquisition zone 1.

[0097] The identification system S also comprises a processing system 24 configured to receive as input the final images generated and to provide as output an identification data item for each food product 5, 6.

[0098] In practice, the processing system 24 comprises, for example, an artificial neural network configured to receive these generated final images as input and to provide the identification data as output. This artificial neural network has previously been trained from known images comprising identified food products. This artificial neural network is, for example, a multi-layer neural network. Preferably, it is a deep multi-layer neural network. It is, for example, a convolutional neural network or a transformer.

[0099] The identification data therefore makes it possible here to identify the food product 5, 6 concerned. For example, this identification data specifies, for the food product 5, that it is a glass filled with cold fruit juice, or for the food product 6, that it is a plate filled with a dish of the day.

[0100] As shown in [Fig. 6], the identification system S finally comprises a remote server SRV. This remote server SRV is for example configured to receive the final images generated (three or four or five depending on the implementation used) by the processing unit 15 of the image acquisition system 10 and the identification data determined by the processing system 24 of the identification system S and transmit this identification data to an external entity 30. This external entity 30 is for example a cashing device making it possible to generate an invoice for the food products concerned present in the acquisition zone 1, for example the payment or billing terminal T shown in [Fig. 1].

[0101] Alternatively, the processing system can be directly integrated into the acquisition system. The identification data is therefore directly determined within the acquisition system.

[0102] [Fig.7] is a flowchart representing an example of the Acq acquisition process image of at least one food product 5, 6 present in the acquisition zone 1 according to a first embodiment.

[0103] This acquisition method Acq is implemented when a user places an item (for example a tray supporting a food product 5, 6) in the acquisition zone 1. The presence of a product 5, 6 in the acquisition zone 1 triggers a command (step E0) to the image acquisition system 10 (and in particular the camera 12, the thermal detector 14 and the three-dimensional depth sensor 16) in order to trigger the acquisition of the different images.

[0104] In practice, the presence of a food product 5, 6 in the acquisition zone 1 is for example detected by a weighing system or by a tray presence detector.

[0105] In step E2, the camera 12 of the image acquisition system 10 receives the command to trigger the image acquisition. It then acquires at least one color image of the acquisition zone 1. For this, the processing unit 15 transmits, for example, a command concerning the exposure duration for the acquisition of the color image. The exposure duration is, for example, here between 12 and 32 milliseconds.

[0106] In practice, the color image is broken down on the basis of three primary colors: red, green and blue (according to the RGB color system or RGB for "red, green, blue" according to the commonly used acronym of Anglo-Saxon origin).

[0107] In this first embodiment, the camera 12 provides an RGB color image (taking into account the three primary colors). More particularly here, the camera 12 provides an RGB color image with the first resolution. In other words, the camera 12 acquires an RGB color image with a high resolution.

[0108] In step E4, the thermal detector 14 of the image acquisition system 10 receives the command to trigger the image acquisition. It then acquires at least one thermal image Ith of the acquisition zone 1.

[0109] In this description, the term "thermal image" means an image in which each pixel is associated with a temperature datum. A scale of colors or shades of colors makes it possible to transcribe the temperature differences in the acquisition zone 1 studied.

[0110] [Fig. 4] represents an example of a thermal image obtained by the thermal detector 14. The temperature scale is represented here in the form of shades of gray (this representation allows in particular an optimization and a reduction of the size of the thermal image in order to improve the storage and transmission of this image). This scale extends here, on the representation of [Fig. 4], between 20 and 55 degrees Celsius (this scale being truncated at 20 degrees for temperatures below 20 degrees and limited to 55 degrees for temperatures above 55 degrees). In other words, here, the temperature data obtained from the thermal detector 14 are processed, by the processing unit 15, according to a bandpass filtering so as to obtain a relevant representation of the acquisition zone 1.

[0111] Preferably, the temperature scale extends between 0 and 60 degrees Celsius. Alternatively, this temperature scale may be wider, depending on the model of the thermal detector used. For example, a thermal detector of the Melexis MLX90640 type can detect temperatures between -40 and 300 degrees Celsius.

[0112] The thermal detector 14 is particularly advantageous for enabling the distinction between food products which visually have the same appearance. For example, an orange juice and an orange-colored soup have the same visual appearance (via for example the images acquired by the camera 12) but different temperatures. The thermal image acquired by the thermal detector 14 therefore makes it possible to distinguish these two products.

[0113] The use of the thermal detector 14 is also advantageous in the case where a food product is masked by a concealing element. This concealing element is, for example, a paper towel or a lid placed on the food product. In this case, the images acquired by the camera 12 do not make it possible to distinguish the food product concerned. The thermal detector 14 makes it possible to highlight the presence of a product by characterizing it by its temperature.

[0114] In step E6, the three-dimensional depth sensor 16 of the image acquisition system 10 receives the command to trigger the image acquisition. It then acquires at least one image of the relief Idpth of the acquisition zone 1. It also acquires another RGB color image of the acquisition zone 1, this other RGB color image having a second resolution. The second resolution is lower than the first resolution (of the RGB color image acquired by the camera 12).

[0115] In this description, the term “relief image” (or image representing the relief) means an image representing depths or reliefs in the acquisition zone, relative to a reference plane. Here the reference plane is for example formed by the laying surface in the acquisition zone 1 on which each serving tray is placed. In the case of a tray used to transport food products 5, 6, the reference plane is for example the plane of this tray.

[0116] [Fig.5] represents an example of a relief image obtained by the three-dimensional depth sensor. More particularly, this figure represents the height H (also called depth in this description) relative to the reference plane defined previously. The depth scale is here represented in the form of shades of gray (this representation allows in particular optimization and a reduction in the size of the relief image in order to improve the storage and transmission of this image). This scale here extends between 19 and 105 millimeters (mm) (this scale being truncated to 19 mm for depths less than 19 mm and limited to 105 mm for depths greater than 105 mm). In other words here, the depth data obtained from the three-dimensional depth sensor 16 are processed, by the processing unit 15, according to a bandpass filtering so as to obtain a relevant representation of the acquisition zone 1.

[0117] Alternatively, other depth scales, in particular wider ones, can be obtained depending on the three-dimensional depth sensor used.

[0118] Advantageously, the three-dimensional depth sensor 16 makes it possible, thanks to the detection of different depths, to improve the recognition of the food products 5, 6. This sensor is particularly useful for enabling the detection of food products, such as bottles or cans, placed in a position preventing their identification (for example in a vertical position for the bottles or cans). This then makes it possible to alert the user in order to allow them to position them in such a way as to make their identification possible (in a horizontal position for example). The three-dimensional depth sensor 16 also makes it possible to detect possible objects hidden in the other images based on a volumetric comparison between an expected volume and the volume observed for an object (for example in the case of a plate or a napkin that is too high).

[0119] According to the example described previously, steps E2, E4 and E6 are implemented simultaneously.

[0120] As shown in [Fig.7], the acquisition method Acq ends with step E8 during which the processing unit 15 of the image acquisition system 10 generates the final images from each image obtained in steps E2, E4, E6.

[0121] As described previously, in this first embodiment, the four final images are formed by a high-resolution RGB color image, a medium-resolution RGB color image, a thermal image Ith and an image of the relief I dpth of the acquisition zone 1.

[0122] The final images generated can then be processed (for example, format processing of the “High Dynamic Range” type according to the commonly used Anglo-Saxon designation of origin or artificial intelligence algorithms allowing image processing) before being used to identify the food product 5, 6. They can also undergo encoding processing, for example in different gray levels.

[0123] The invention also relates to a second embodiment of the method for acquiring Acq an image of at least one food product 5, 6 present in the zone acquisition 1.

[0124] In this second embodiment, the image acquisition method Acq has the same steps and characteristics as the first embodiment described previously.

[0125] The only difference lies in the images obtained at the output of the camera 12. According to the second embodiment, at the output of the camera 12, three images are obtained, one in each primary color.

[0126] Thus, during step E8 during which the processing unit 15 of the image acquisition system 10 generates five final images from each image obtained in steps E2, E4, E6, the five final images obtained are formed by three color images, each in a primary color, a thermal image Ith and an image of the relief Idpth of the acquisition zone 1.

[0127] The invention also relates to a third embodiment of the method for acquiring Acq an image of at least one food product 5, 6 present in the acquisition zone 1.

[0128] In this third embodiment, the image acquisition method Acq has the same steps and characteristics as the first embodiment described previously.

[0129] The only difference lies in the outputs obtained from the image acquisition system. According to the third embodiment, five final images are obtained: an RGB color image, an underexposed image, an overexposed image, a thermal image and an image of the relief of the acquisition area. In this third embodiment, the overexposed image is obtained using a long exposure time and the underexposed image is obtained using a short exposure time. The RGB color image then corresponds to an average exposure compared to the other two images (underexposed and overexposed).

[0130] Thus, during step E8 during which the processing unit 15 of the image acquisition system 10 generates five final images from each image obtained in steps E2, E4, E6, the five final images obtained are formed of an RGB color image, an underexposed image, an overexposed image, a thermal image and an image of the relief of the acquisition zone.

[0131] The invention also relates to a fourth embodiment of the method for acquiring Acq an image of at least one food product 5, 6 present in the acquisition zone 1.

[0132] In this fourth embodiment, the image acquisition method Acq has the same steps and characteristics as the first embodiment described previously.

[0133] The only difference lies in the images obtained at the output of camera 12. According to the fourth embodiment, during step E8 during which the processing unit 15 of the image acquisition system 10 generates the final images from each image obtained in steps E2, E4, E6, four final images obtained are formed by two RGB color images, a first according to a first viewing angle and a second according to a second viewing angle, a thermal image Ith and an image of the relief Idpth of the acquisition zone 1.

[0134] The invention also relates to a fifth embodiment of the method for acquiring Acq an image of at least one food product 5, 6 present in the acquisition zone 1.

[0135] In this fifth embodiment, the final images are three in number and formed of an RGB color image, a thermal image Ith and an image of the relief Idpth of the acquisition zone 1.

[0136] Whatever the embodiment of the image acquisition method, certain execution variants can be envisaged.

[0137] According to an alternative embodiment, in step E2, the trigger command for the image acquisition system can be issued following a manual action, for example by the user who has just placed his tray in the acquisition zone.

[0138] According to another variant embodiment, steps E2, E4 and E6 can be implemented successively in a short time interval, for example less than 5 milliseconds.

[0139] The invention then also relates to a method of identifying Id at least one food product 5, 6 placed in the acquisition zone 1. This method of identifying Id follows the method of acquiring images Acq described previously. [Fig.8] represents, in the form of a flowchart, an example of an identification method Id implemented according to the invention.

[0140] The identification method Id firstly comprises a step Acq of acquiring a plurality of images of each food product 5, 6 present in the acquisition zone 1 according to the acquisition method Acq described previously.

[0141] The identification method Id then comprises a step E10. During this step, the processing system 24 of the identification system S identifies each food product 5, 6 by providing identification data associated with each food product 5, 6 from the final images generated (three or four or five depending on the implementation used) during the acquisition step Acq. As described previously, this step is for example implemented by means of an artificial neural network receiving, as input, the final images generated (three or four or five depending on the implementation used) and providing, as output, the identification data associated with each food product 5, 6.

[0142] Each identification data obtained is for example then transmitted to an entity external, such as a cash register device to bill the user for the products they have chosen on their tray.

[0143] Advantageously according to the invention, the use of the thermal detector 14 also makes it possible to carry out a quality control of the food products positioned in the acquisition zone, thanks to the known temperature data. The processing system 24 is then able to associate and transmit (to an external entity), for each food product, the associated identification data and the corresponding temperature data.

[0144] The invention also finds a particularly advantageous application in the estimation of the nutritional and caloric characteristics of foods. For this, each identification data obtained is for example analyzed from a nutritional and caloric point of view, by comparison, for example, with data tables listing these characteristics for the different foods.

[0145] The image acquisition system 10 also finds a particularly advantageous application for quantifying the portions of food remaining after the meal. For this, the acquisition zone is, for example, a space for placing a tray after the meal. The image acquisition system operates in the same way as described previously to acquire the final images (three or four or five depending on the implementation used) of the acquisition zone after the meal.

[0146] Then, a processing system identifies each product remaining on the tray by providing another identification data item associated with each remaining product from the final images generated. Here too, this step is for example implemented by means of an artificial neural network receiving, as input, the final images generated and providing, as output, the other identification data item associated with each remaining product.

[0147] Each other data obtained is for example then transmitted to an external entity, such as an analysis device making it possible to evaluate the quantity of food remaining.

[0148] Thus, advantageously according to the invention, on the basis of the images acquired by the image acquisition system, it is possible to analyze the quantities of food remaining and therefore to quantify food waste, for example in a collective catering service.

Claims

Claims

1. System (S) for identifying at least one food product (5, 6), comprising an acquisition zone (1) adapted to receive at least one food product (5, 6), a system (10) for acquiring a plurality of images of each food product (5, 6) and a processing system configured to provide identification data for each food product present in the acquisition zone (1), the acquisition system comprising: - a camera (12) configured to acquire at least one color image of the acquisition zone (1), the camera (12) having a first optical axis (A1); - a thermal detector (14) configured to acquire at least one thermal image (Ith) of the acquisition zone (1), the thermal detector (14) having a second optical axis (A2);- a three-dimensional depth sensor (16) configured to acquire at least one image of the relief (Idpth) of the acquisition zone (1), the three-dimensional depth sensor (16) having a third optical axis (A3), the thermal detector (14) being positioned between the camera (12) and the three-dimensional depth sensor (16), the first optical axis (Al), the second optical axis (A2) and the third optical axis (A3) being parallel to each other, the first optical axis (Al) and the third optical axis (A3) being spaced apart by a distance (d) of between 5 and 20 centimeters; - a processing unit (15) configured to receive each image from the camera (12), the thermal detector (14) and the three-dimensional depth sensor (16);characterized in that the processing unit (15) is configured to generate a final image from each received image, and in that each final image is transmitted to a processing system (24) comprising an artificial neural network, and in that the artificial neurons are configured to identify each food product (5, 6) from the final images.;

2. Identification system (S) according to claim 1, characterized in that the artificial neural network is a multi-layer neural network.

3. Identification system (S) according to claim 1 or 2, characterized in that the artificial neural network is a convolutional neural network.

4. Identification system (S) according to one of claims 1 to 3, characterized in that the acquisition system (10) comprises communication means (18) connected to the processing unit and configured to exchange data with a remote server (25).

5. Identification system (S) according to claim 4, characterized in that the communication means (18) are configured to transmit the final images to the remote server (25).

6. Identification system (S) according to claim 4 or 5, characterized in that the communication means (18) are capable of transmitting to a cash register device (30), the identification of the food products coming from the processing system (24).

7. Identification system (S) according to claim 6, characterized in that the cashing device (30) is capable of generating an invoice, from the identification of the food products coming from the processing system (24).

8. Identification system (S) according to one of claims 1 to 7, characterized in that the acquisition system (10) is positioned at a distance (h) of between 50 and 90 centimeters from the acquisition zone (1).

9. Identification system (S) according to one of claims 1 to 8, characterized in that the processing unit (15) is configured to obtain two distinct color images of the acquisition zone (1), respectively according to a first resolution and a second resolution, four final images being formed from a color image with a first resolution, a second color image with a second resolution, the thermal image (Ith) and the relief image (Idpth) of the acquisition zone, the first resolution being greater than the second resolution.

10. Identification system (S) one of claims 1 to 9, characterized in that the processing unit (15) is configured to obtain three distinct color images of the acquisition zone (1) each obtained respectively according to a first exposure, a second exposure and a third exposure, five final images being formed of a color image in each of the first exposure, the second exposure and the third exposure, the thermal image (Ith) and the relief image (Idpth) of the acquisition zone.

11. Identification system (S) according to one of claims 1 to 10, characterized in that the color image is decomposed into three primary colors, five final images are formed from one image in each of the three primary colors, the thermal image (Ith) and the relief image (Idpth) of the acquisition area.

Images