Computer-assisted method for processing radar data in a radar-assisted retail system

Abstract

The invention relates to a computer-assisted method for processing radar data in a radar-assisted retail system, the method comprising the following method steps: continuously outputting raw radar data from each radar sensor of multiple radar sensors installed in a salesroom, with the raw radar data representing a situation, detected by means of radar, in a relevant radar-sensor detection range of the radar sensor in question; continuously analyzing the raw radar data for the purpose of generating description data, which description data digitally describe the relevant detected situation in terms of at least the presence of a person in the radar-sensor detection range, preferably also in terms of the location of the person in the radar-sensor detection range if their presence has been detected; and providing the description data as such and / or a summary of the description data via a programming interface.

Description

[0001] Computer-aided method for processing radar data in a radar-assisted retail system

[0002] Description

[0003] Technical field

[0004] The invention relates to a computer-aided method for processing radar data in a radar-supported retail system.

[0005] The invention further relates to a radar-assisted retail system for carrying out the computer-assisted method.

[0006] background

[0007] WO 2023 / 021419 A1 discloses a system for tracking objects at a retailer. It uses a network of radar sensors distributed throughout a monitored area of ​​the retailer, which directly exchange metadata about an identified object in order to track that object.

[0008] However, this type of radar sensor deployment has proven extremely disadvantageous—particularly when implemented with battery-powered radar sensors. This is because the radar sensors not only have to detect the object, but also analyze and characterize the radar data generated during object detection in real time to describe it with metadata. This requires considerable local computing power and processing speed. This metadata must then be exchanged with each other in real time to enable object tracking across radar sensors. Both of these processes consume considerable electrical power. Furthermore, each radar sensor must have a powerful computer architecture to handle real-time processing of the radar data so that the metadata is also available in real time to neighboring radar sensors for object tracking.The invention therefore has for its object to provide an improved method for processing radar data in a radar-assisted retail system which overcomes the problems mentioned.

[0009] Summary of the invention

[0010] This object is achieved by a computer-assisted method according to claim 1. The subject of the invention is therefore a computer-assisted method for processing radar data in a radar-supported retail system, wherein the method comprises the following method steps, namely a continuous output of radar raw data from one radar sensor of a plurality of radar sensors installed in a sales area, wherein the radar raw data represent a radar-detected situation in a respective radar sensor detection range of the respective radar sensor, a continuous analysis of the radar raw data for the purpose of generating description data, which description data digitally describe the respective detected situation with regard to at least the presence of a person in the radar sensor detection range, preferably also with regard to the location of the person in the radar sensor detection range if presence is detected,and providing the description data as such and / or a summary of the description data via a programming interface.

[0011] This object is further achieved by a system according to claim 25. The subject matter of the invention is therefore a radar-assisted retail system designed to carry out the method according to the invention.

[0012] The measures according to the invention offer the advantage that the analyzed raw radar data for each radar sensor, i.e., either the descriptive data obtained from it or the summary of the descriptive data generated from it, are not immediately disseminated for a specific purpose, but are initially made available for further processing as flexibly and accurately as possible. This provision is thus sensor-specific for each radar sensor and subsequently allows for the most flexible and needs-oriented use possible.

[0013] This provision is achieved via the programming interface. This programming interface enables defined access to the description data—if applicable, and / or to its summary—by other programs and / or data processing devices, collectively referred to as querying or accessing instances. This type of provision creates the basis for diverse processing of the description data, adapted to specific needs. This further processing can now be defined with complete flexibility by internal system components assigned to the retailer's system or by an external system assigned to another market participant granted access to the programming interface.

[0014] In general, an instance is a computer executing a program or process to perform a specific task; in this context, this task consists of accessing or querying the programming interface. Such an instance can be physical, i.e., formed by a dedicated computer, or virtual, i.e., purely software-based, deployed on a physical computer or in the cloud.

[0015] Further, particularly advantageous embodiments and developments of the invention emerge from the dependent claims and the following description.

[0016] The description data itself indicates whether at least one person is present within the radar sensor's detection range or whether the radar sensor's detection range is clear. If the presence of at least one person has been detected within the radar sensor's detection range, the description data also indicates the person's location within the radar sensor's detection range. The same applies if the presence of multiple people has been detected within the radar sensor's detection range; in this case, the description data indicates the respective location of the person in question within the radar sensor's detection range.For example, the summary of the description data for the radar sensor in question, in whose radar sensor detection area the presence of at least one person was detected, can indicate how many people are currently in the radar sensor detection area, i.e. indicate the number of people.

[0017] Deriving from the fact that a standstill is determined for the respective person, which in turn is individually represented by the descriptive data, the summary of the descriptive data can also provide information regarding the length of stay of the person(s).

[0018] Deriving from the fact that a movement is detected for the respective person, which in turn is individually represented by the description data, the summary of the description data can also contain information regarding the movement of the person(s), such as a direction indication, a speed or an acceleration or a recognized - possibly predefined - movement pattern.

[0019] All of this information, represented by the descriptive data or its summary, can refer to the entire radar sensor detection range or only to a sector of the radar sensor detection range. If sector-specific processing is intended, the sector of interest is selected or specified by software.

[0020] Regarding the radar sensor used, it should be noted that this could, for example, be a fixed-radar installation, which, however, entails a relatively complex and compact antenna configuration that would need to be installed in a suitable location. A radar sensor with a rotating antenna—e.g., installed on the ceiling of a room—could also be provided.However, it has proven particularly advantageous if the radar sensor is designed as a frequency modulated continuous wave radar (FMCW radar for short), and the modulated continuous wave radar outputs a digital representation of its received signal as the radar raw data, comprising an in-phase component signal (I signal for short) and a reactive component (Q signal for short). The ongoing analysis is performed directly on the basis of the in-phase component and the reactive component. These signals are collectively referred to as IQ signals. Such a radar sensor is characterized by its extremely compact design. In this embodiment, the radar sensor is implemented as an integrated circuit.Depending on the antenna configuration used, several objects, specifically their distance positions, can be detected in one detection pass.

[0021] The radar sensor's integrated circuit can, for example, have its own internal processing unit (e.g., an Application Specific Integrated Circuit) with the help of which very rudimentary information (such as the detection of an object, the location of the detected object, or even just the detected movement of the object) regarding the detected situation can be provided. However, using this functionality integrated in the radar sensor's integrated circuit limits the information content regarding the detected situation provided by the radar sensor. This circumstance was recognized by the applicant. Therefore, the IQ signals provided by the radar sensor are tapped directly, i.e., the analysis of the I and Q signals preconfigured in the radar sensor circuit is left unused or bypassed, and a continuous analysis of the IQ signals is carried out, defined and implemented by the applicant.For this application, the radar sensor is designed to provide its IQ signals using a digital data stream.

[0022] According to this aspect, the ongoing analysis of the radar raw data is now carried out with the help of a data processing module connected to the respective radar sensor. The data processing module receives the digital data stream representing the IQ signals and processes the IQ signals. The data processing module can be a small, standalone computer, such as a microcontroller or a microprocessor, which can be equipped with appropriate peripheral components. Its programming allows for the diverse aspects of the ongoing analysis of the IQ signals that are appropriate to the respective application environment, which, for example, are completely ignored in the pre-assembled integrated circuit of the radar sensor.

[0023] In this context, it has proven particularly advantageous that the respective radar sensor and the data processing module connected to it form a radar assembly, in particular a radar assembly integrated into a system component of the retail system.

[0024] On the one hand, this allows the individual installation of the radar module, detached from other system components of the retail system, directly in a shelf or on a shelf rail, etc., so that the radar module can be used as an independent system component - if necessary with its own housing and corresponding fastening device or for fastening suitable training - of the retail system in the business area (sales outlet or warehouse).

[0025] On the other hand, the radar assembly can also be integrated into a system component - i.e. an electronic device - of the retail system, such as an ELS (Electronic Shelf Label) or a camera, etc. Integrated into this system component, the radar assembly can then be used in the retailer's sales premises or in a retailer's storage area, depending on the positioning typically assigned to this system component. Such an integration of the radar sensor also has the advantage that its presence in the sales premises is not noticed by customers or staff, which eliminates any possible reservations that might arise against the use of such a radar sensor. Such a radar assembly also enables the respective situation to be recorded from the position orthe orientation of the relevant system component into which it is integrated. This also has the advantage that the orientation of the radar module, in particular of the radar sensor detection area, is automatically defined by the orientation of the system component. Advantageously, the direct analysis of the IQ signals can be implemented in a completely customized way and can therefore also take individual aspects into account. For example, the data processing module can apply at least one object classification model and / or at least one object prediction model during the ongoing analysis. These software-based models can be individually adapted to the respective area of ​​application for the entirety of the radar modules used, for example, in the customer area of ​​the retailer, for a selection of the radar modules used, or even individually for individual radar modules.For example, one model may be optimized for use in the fruit and vegetable shelves, whereas another model may be optimized for use in the specialty display cases, and yet another model may be optimized for use at the end of aisles or on a presentation shelf or on a shopping cart. The models allow an accurate selection of objects of interest from all the detected objects in the scene within the radar sensor's detection range, tailored to the respective detection situation, i.e. the respective location and the typical topology present there. This way, people in particular can be reliably detected in relation to shopping carts, shopping baskets or other detected structures, such as shelves or sales tables or sales baskets, as well as the goods presented there, etc. It is also possible to accurately distinguish between static and moving orpotentially moving objects (e.g., people) that are currently at rest against a static background. The models can also be optimized to categorize the detected people based on their size, thus enabling a rough age estimate.

[0026] In this context, it has proven particularly advantageous that the data processing module has a machine learning mode, and in the machine learning mode, the raw radar data is used to create the at least one object classification model and / or the at least one object prediction model. This allows the respective model to be used to be created in real time in the real environment or with the aid of the raw radar data that was previously acquired using an equivalent sensor or the same sensor in the real environment, saved, and then used as training data for training. A radar assembly prepared in this way, i.e. adapted to its actual location of use, delivers significantly better results than would be expected from a preconfigured radar sensor with its integrated functionality.However, in order to create the respective model, a computerized simulation of the radar assembly can also be used, whereby the model is then created on a computer using the aforementioned training data and is then imported into the relevant data processing module in digital form in order to apply it in the real environment.

[0027] According to a further aspect relating to the use of the description data or their summary, it is advantageously provided that the programming interface is contacted by a querying instance and, depending on the description data or their summary available via the programming interface according to the query, a behavior of a system component of the retail system is controlled and / or data provided by the system component is interpreted.

[0028] Based on the descriptive data or its summary, i.e., depending on it or the associated information content, the behavior of the system component can be directly influenced. For example, data collection by the system component or even the manner in which data is provided by the system component can be directly influenced, or even completely prevented.

[0029] Analogously, the system component can be directly influenced to determine how or whether data received by the system component should be processed.

[0030] This allows for a direct impact on energy consumption, as less processing activity also results in less energy consumption. Local energy consumption at each system component can thus be controlled so that it only occurs when absolutely necessary. With a significant number of such system components, which can easily amount to several thousand in a retailer's premises, this measure unlocks significant energy savings potential.

[0031] This measure can also reduce the volume of data traffic in the retailer's system to an absolute minimum.

[0032] The only prerequisite for this measure is that the respective system component is designed to be controllable.

[0033] On the basis of the descriptive data or their summary, i.e. depending on this or the associated information content, it is also possible to intervene in the interpretation of the data provided by the system component. In this case, the system component can continue to work unaffected and the question of whether and to what extent the data it provides is further processed is decided by another data processing device (at another location within or outside the system) or by another software, taking the descriptive data or their summary into account. In this case, system components can be used that do not have to be controllable. In addition, this measure has the effect that the system components can continue to work unaffected by the descriptive data orwhose summary provides its data, and a first data processing device or a first software instance decides, depending on the description data or its summary, whether and / or to what extent this data is to be processed. At the same time, i.e., in parallel, the data supplied by the system component can be processed by a second data processing device or a second software instance without being influenced by the description data or its summary.

[0034] Specific examples will be discussed below. Furthermore, it has proven advantageous to identify the radar raw data with a radar sensor identifier, which allows for a clear identification of the radar sensor from which the radar raw data originates, and to identify the description data with the radar sensor identifier. The radar sensor identifier can, for example, be embedded in the data stream of the IQ signals of the respective radar sensor or be provided additionally. The radar sensor identifier can also be formed by an identifier of the radar module, which also allows for a clear - direct - identification of at least the radar module and - at least indirectly - the radar sensor installed in it. Using the radar sensor identifier, a location reference can be established for the respective radar sensor, such asvia a data structure that stores the location reference, e.g. in a business area of ​​the retailer, for each radar sensor used, specifically its radar sensor identifier, or the radar assembly that has the radar sensor, or the system component that has the radar assembly. This location reference can be given by specifying the absolute - e.g. three-dimensional - coordinates in the business area or by specifying another structure that has the radar sensor or radar assembly, such as a shelf, a sales counter or a display case, etc., where the position of this other structure in the business area is known. The data structure can be stored in a general database or form part of a plangram or a realogram of the business area.

[0035] According to a first embodiment, the system component can be formed by a camera which is designed to deliver a digital image of a scene in its camera detection area, and wherein with the aid of the querying instance, if the description data or their summary indicate that a radar sensor detection area which also covers the camera detection area is free of persons,

[0036] - the camera is at least temporarily caused to record the scene or

[0037] - the digital recording of the scene provided by the camera is marked as person-free.

[0038] This measure takes data protection aspects into account and reliably prevents images of people from being stored and further processed without their consent, which would typically be the case in a retailer's business premises. According to the first variant, recordings by the camera are only permitted if the camera's detection area is clear of people. This influences the camera's detection function.

[0039] According to a second variant, camera recordings that may contain people can be specifically excluded from further processing and storage if they are not marked as person-free.

[0040] This measure, for example, can be used to monitor a shelf with a camera and digitally record the product levels. Advantageously, only those images that are truly free of people are sent to resource-consuming analysis software, allowing for a clear and accurate assessment of inventory levels. This avoids, in particular, processing problems caused by people in the scene and captured in the image, who shadow or obscure the shelf behind it.

[0041] Alternatively - i.e., in contrast to the determination that the detection area is free of persons - it can also be provided that with the help of the querying instance, if the description data or their summary indicate that at least one person is present in the radar sensor detection area,

[0042] - the camera is prevented from recording the scene at least while the person is present, or

[0043] - the digital image provided by the camera is marked as containing at least one person. This form of implementation also results in the aforementioned effects, whereby the information obtained from the query result from the programming interface must be implemented accordingly.

[0044] According to a further embodiment, it can also be provided that the system component is formed by a display device << ESL with LED / Hermes Rail with LED / Video Rail / Video Display >> which is located within the camera detection range, and wherein with the aid of the querying instance, under the condition of the person-free radar sensor detection range, the display device is caused to emit a light signal which can be detected with the aid of the camera.

[0045] The display device can be an electronic shelf label (known in technical jargon as an "Electronic Shelf Label", or ESL for short), which has a light-emitting diode (LED for short) for emitting the light signal. It can also be an electronic shelf rail which has such an LED. Furthermore, it can be a video shelf rail or generally an electronic display label which has this LED. The LED can be used to more easily identify the respective device in a camera recording. This is used, for example, by a retailer to find the respective electronic display label, whose LED is flashing, in the digital image of the camera, to display its display content (such asThe LEDs are designed to recognize the unique identifier of the display sign and the associated product, and then to logically link the product to which the electronic display sign is assigned with the display sign, which is referred to in technical jargon as "binding," in order to subsequently supply it with the product-specific data for the purpose of displaying the same. Operating the LED is a power-consuming activity, particularly in battery-operated display signs, which has a detrimental effect on battery life. But even with an external power supply, such as a power pack, flashing represents a significant power consumption.

[0046] Therefore, the measure that the light signal is only emitted when the radar sensor detection area is clear of people represents an essential measure for reducing energy consumption. At the same time, in combination with the previously discussed measures, it ensures that the LED only lights up when the camera, which is also controlled based on the descriptive data or its summary, is authorized to detect the light signals, or when the data emitted by the camera can be further processed because it is marked as clear of people. Furthermore, the measure ensures that customers moving around the store are not irritated by light-emitting LEDs if they are currently moving or staying within the radar sensor's detection area. However, the power-saving effect of electronic consumers cannot only be applied to the LEDs of electronic display signs.Rather, this aspect can also be utilized with respect to lighting devices in the business area. In this context, it has proven advantageous for the system component to be formed by a lighting device that is adaptable with regard to its light output, which is provided for illuminating a radar sensor detection area and / or an object present in the radar sensor detection area, or for emitting light from an object located in the radar sensor detection area. With the aid of the control device, if the descriptive data or their summary indicate a radar sensor detection area free of persons,

[0047] - the light emission from the lighting device is prevented or

[0048] - the light output caused by the lighting device is changed at least in comparison to a light output which is set when a person is present in the radar sensor detection area.

[0049] The adaptability of the light output can be achieved, for example, by a design to change the light intensity ("dimming") or by a design to change the color ("controlling an RGB LED") of the emitted light or by a design to change the signaling carried out with the help of the light ("continuous lighting with constant intensity", "intensity modulation", "pulse width modulation").

[0050] The lighting device can, for example, be a light source for illuminating objects or locations. The lighting device can also be intended to illuminate the interior of a cabinet, such as a refrigerator or refrigerated display case, or the like. However, the lighting device can also be the backlight of an electronic display. The lighting device itself can also form an entire electronic display. Provided the radar sensor detection area is clear of people, the lighting device can, for example, be completely switched off to avoid any power consumption, or at least be switched to energy-saving standby mode.

[0051] If the light emission is not to be completely eliminated, the light emission set when a person is present can be changed to a light emission with reduced power consumption, provided that the radar sensor detection area is clear of people.

[0052] According to a further embodiment, the system component can be formed by a display device which is designed to display a content which is perceptible to humans within a perception range, and wherein with the aid of the control device,

[0053] - if the description data or its summary indicates a person-free radar sensor detection area covering the perception area, the display device is caused to display static content, and,

[0054] - if the descriptive data or its summary indicates the presence of a person in the radar sensor detection area, the display device is caused to display dynamic content, or vice versa.

[0055] This measure allows you to switch the type of content, either from static content, such as a still image or simply a color image (the screen lights up red or green or blue, etc.), to dynamic content, such as a sequence of still images or the playback of a video, or alternatively from dynamic content to static content.

[0056] If the static content is played when no person is present, this results in an energy-saving effect because this type of content requires fewer resources and computing power. However, from a marketing perspective, it may be preferable that no video is played when a person is present, because the person should focus on the static content. In this case, the energy-saving effect is deliberately foregone in favor of the marketing effect. According to a further embodiment, the system component can be formed by a sensor different from the radar sensor, which sensor detects a physical parameter such as humidity, brightness, or temperature, etc., and with the help of the querying instance, if the descriptive data or its summary indicates that a person is present in the radar sensor's detection range,

[0057] - the detection of the physical parameter by the sensor is at least temporarily suspended or

[0058] - Detection data representing the detected physical parameter and emitted by the sensor are marked as having been recorded in the presence of a person.

[0059] This measure has the effect of reliably preventing any possible distortion of the sensor's detection result caused by the presence of the person, such as shading of the sensor from ambient light or an increase in humidity caused by a perspiring person or an increase in the ambient temperature caused by the presence of the person, etc.

[0060] Aspects of the programming interface are discussed below. In this context, it has proven particularly advantageous that the programming interface generates the summary of the description data depending on the parameterization transmitted to it. The programming interface is therefore configured or programmed in such a way that it can receive and implement the parameterization for the summary from an external software application. Its functionality can therefore be freely configured and situation-specifically within the framework of the specified and permissible parameterization.

[0061] In this context, it has also proven advantageous for the parameterization to specify at least one of the radar sensors for which the summary of the description data is to be generated according to the parameterization, and for the programming interface to apply the parameterization to the description data generated from the raw radar data of the specified radar sensor. This measure ensures that the querying instance accessing the programming interface can select precisely the sensor or sensor group required for its purposes. This allows the specific radar sensors to be selected whose raw radar data is to be submitted for further analysis. In this context, the querying instance can have access to the aforementioned data structure in which the location reference for the respective radar sensor is stored.The querying entity may also have access to the planogram or realogram to identify those radar sensors that are of interest in the context of the planogram or realogram.

[0062] With regard to the summary of the descriptive data, it has proven particularly advantageous if

[0063] - the parameterisation specifies a temporal aspect and / or a quantitative aspect of the radar-detected situation, and

[0064] - the programming interface summarizes and provides a sequence of the description data generated from the radar raw data of at least one of the radar sensors with regard to the specified temporal aspect and / or with regard to the specified quantitative aspect.

[0065] This allows the radar-detected events within the radar sensor detection range to be evaluated in a variety of ways according to the parameterization and provided as a summary. For example, it is possible to determine how long people stayed within the radar sensor detection range, how many people were present within the radar sensor detection range during a certain period of time, and how many of these people actually stayed within the radar sensor detection range during the specified period of time for a duration that exceeded a predefined threshold. These summaries can be further processed for marketing purposes and provide the basis for new business models.

[0066] The summary of descriptive data can be performed individually for each radar sensor. However, it can also be planned to combine the descriptive data from multiple radar sensors in order to make a comprehensive statement across different radar sensor detection areas. This is particularly useful, but not mandatory, for adjacent or slightly overlapping radar sensor detection areas.

[0067] According to a further aspect, the summary of the description data can be provided together with event data describing an event generated by the retail system, the occurrence of which corresponds to the summary of the description data. Linking the summary of the description data with the corresponding events in the retail system can provide information about the success or failure of a marketing measure or the effect of operational measures, which can be evaluated by a higher-level data processing device or software instance.

[0068] For example, the event can represent the status of the electrical power management in the store. However, an event can also represent an external trigger, such as a marketing campaign by the retailer's competitors or a weather report captured via an internet interface. An event can also represent a video played on a screen in the store. In more comprehensive terms, an event can also represent a playlist. In its simplest form, however, an event can specify a defined time period that is of interest for analysis. Optionally, multiple time periods can be specified, or periodically recurring time periods can be defined as events.

[0069] According to a further aspect, it may also be advantageous if the summary of the descriptive data is supplemented by supplementary data, whereby the supplementary data is obtained from a data source other than one of the radar sensors. This measure enables, for the first time, the summary of the descriptive data to be evaluated in light of the supplementary data, forming the technical basis for a further business model.

[0070] In this context, it has proven particularly advantageous if the supplementary data represents at least one of the following information: - the number of QR code scans (QR stands for "Quick Response Code", with reference here to the ISO / IEC 23941:2022 standard as an example),

[0071] - the number of web link views,

[0072] - the number of NFC interactions (NFC stands for "Near Field Communication" as defined by the ISO / IEC 18092 / ECMA-340 or ISO / IEC 21481 / ECMA-352 standards),

[0073] - the number of units of a product sold

[0074] - the total sales volume of a product sold.

[0075] The number of QR code scans refers to interactions between people who are within the radar sensor detection range, for example while scanning the QR code of a product that is assigned to the radar detection range. The same applies to NFC interactions, e.g. with an NFC tag or an NFC-equipped ESL. By scanning the QR code or NFC interaction with the person's smartphone, a web link can be transferred to the person's smartphone. Therefore, the number of web links accessed on the people's smartphones to the product or the service offered there can also be represented by the supplementary data. With a time delay to the occurrence of the summary of the descriptive data, the number of units sold of a product that is assigned to the relevant radar sensor detection range or the resulting sales total can also be represented by the supplementary data.

[0076] Particularly preferably, the supplementary data are generated for a time period defined by the parameterization. The specification of a time period is of interest because a marketing campaign is typically defined for an observation period. Programming the application programming interface to allow parameterization in the form of a time period thus forms the technical basis for the time-period-based evaluation of the summary of the descriptive data in light of the supplementary data available for this time period.

[0077] Preferably, the summary of the descriptive data is provided together with the supplementary data. This provision is preferably carried out via a cloud solution, in which the coincident pairs of summary of the descriptive data and supplementary data are made available for further processing, particularly by third parties, such as marketing agencies or product providers or producers. Outside the retailer's sphere of data control, these third parties can thus conduct their individual evaluation of marketing measures (marketing campaigns) and adapt them accordingly.

[0078] To make the process as versatile as possible and accessible to different stakeholders, it has proven particularly advantageous for the programming interface to handle multiple parallel accesses by different software applications (especially by different querying instances). For example, the retailer can check internally whether energy-saving measures are achieving their desired effect. Independently of this, or in parallel, the visually perceptible content provided by the retailer on the various screens can be influenced depending on whether people are present or not. Again, independently of this, the previously discussed camera-supported "binding" can be implemented while observing data protection aspects.Independently of this, third parties who have been granted access to the programming interface can use the aforementioned analyses and evaluations to evaluate the marketing measures.

[0079] The programming interface can, for example, be provided by the retailer's data processing device. In the simplest form, this can be done using a computer or a server or the like. According to a preferred embodiment, the programming interface itself is hosted in a cloud. According to this solution, the retailer's IT system only provides the descriptive data of the individual radar sensors, possibly with a location reference or a product reference that indicates which location or area in the store or which product the respective radar sensor is assigned to. All further processing measures are then provided in the cloud, i.e., with the help of an external data center.This allows for the most flexible and flexible connection possible to a wide variety of other IT systems, such as the retailer's technical operating system, which operates the various electronic displays, controls the cameras, processes their recordings, or manages the retailer's energy consumption. At the same time, third-party IT systems can also access the cloud-hosted programming interface for the purposes described above, with appropriate access rights.

[0080] Finally, it should be generally mentioned that the electronic devices discussed (ESLs, smartphones, tablet computers, video shelf rails, etc.) naturally contain electronics. The electronics can be discrete or integrated, or even a combination of both. Microcomputers, microcontrollers, and Application Specific Integrated Circuits (ASICs) can also be used, possibly in combination with analog or digital electronic peripheral components. Many of the device functionalities mentioned are implemented – possibly in conjunction with hardware components – with the help of software running on an electronic processor. Devices designed for radio communication usually have an antenna configuration for transmitting and receiving radio signals as part of a transceiver module.The electronic devices can also have an internal electrical power supply, which can be implemented, for example, with a replaceable or rechargeable battery. The devices can also be powered wired, either via an external power supply or via "Power over LAN."

[0081] These and other aspects of the invention are apparent from the figures discussed below.

[0082] Short character description

[0083] The invention is explained in more detail below with reference to the accompanying figures using exemplary embodiments, to which, however, the invention is not limited. In the various figures, identical components are provided with identical reference numerals. They show schematically: Fig. 1 shows a section of a retail system according to the invention with which a method according to the invention is carried out;

[0084] Fig. 2 shows a radar assembly used in the system;

[0085] Fig. 3 a camera used in the system with such a

[0086] radar assembly;

[0087] Fig. 4. a visualization of description data BD generated by the radar assembly and summaries of the description data generated therefrom;

[0088] Fig. 5 shows, in block diagram form, functional blocks of a programming interface for generating summaries of the description data as a function of parameterization by a querying instance;

[0089] Fig. 6 a central role of the programming interface in interaction with a large number of querying instances;

[0090] Fig. 7 shows a flow chart of central process steps of the process according to the invention.

[0091] Description of the embodiments

[0092] Figure 1 shows a section of a retail system 1, hereinafter referred to as system 1 for short, with the aid of which a method 44 according to the invention is discussed. The system 1 has a local server 2, with the aid of which a first software (or software instances) for operating or controlling in-house electronic system components, which will be discussed below, is processed in the retailer's business premises. The server 2 is connected via a network 3 to the in-house system components on the one hand and to an external cloud 4 on the other. The term cloud 4, often also referred to as cloud computing or computer cloud or data cloud, refers to a data processing operation that accesses computer resources for data processing, usually via the Internet. A second software (or software instances) is run on these computer resources hosted outside the business premises.software instances), which are discussed in more detail below.

[0093] In simplified form, a shelf 5 and a shopping cart 6 are visualized in the retailer's business area for displaying goods that are not shown for reasons of clarity. A person 7 pushing a shopping cart 8 moves between shelf 5 and shopping cart 6.

[0094] A first light 10 is installed on the ceiling 9 of the store to illuminate the aisle along the shelf 5. The first light 10 is powered by mains power and is designed to be wirelessly controlled, allowing it to be switched on or off or dimmed wirelessly.

[0095] Above the shopping cart 6, a second light 12 is positioned on a mounting rod 11 to illuminate the goods presented in the shopping cart 6. It is also powered by mains power and is designed to be wirelessly controlled, allowing it to be switched on and off, or dimmed, via wireless.

[0096] Furthermore, a battery-operated camera 13 is mounted on the ceiling 9. Its optical camera detection range, symbolically delimited by the boundary lines 13A and 13B, is aligned or adjusted such that the camera 13 captures an image of the entire shelf 5. The camera 13 is designed for digital image capture and for wirelessly transmitting recording data representing the scene within the camera detection range. This allows all objects attached to or placed on the shelf 5 to be captured. For this purpose, the camera 13 is positioned centrally to the shelf 5 at a distance slightly above the shelf 5 and can thus also capture the person 7 if they move or stand in front of the shelf 5, as shown here.

[0097] Furthermore, various system components for displaying product and / or price information or further information relating to marketing measures, such as advertising messages, discount information, etc., are mounted on the shelf 5 as well as the shopping cart 6, which will be discussed in detail below.

[0098] Specifically, the system components mounted on the shelf 5 and the shopping cart 6 are battery-operated electronic price displays 14, also referred to in technical jargon as "electronic shelf labels" and subsequently abbreviated as ESL 14. They are equipped with energy-saving electronics, in particular with an electrophoretic screen 15, primarily to display product and / or price information. The ESLs 14 have an ESL controller (not visualized in detail) designed for radio reception of the product and / or price information and controlling the screen 15.

[0099] A battery-operated electronic shelf rail 16 is installed on the shelf 5 as a further system component, extending approximately halfway along the length of the shelf 5. Electronic display signs 17, which are also equipped with the electrophoretic screen 15, are attached to the electronic shelf rail 16. The display signs 17 are supplied with electrical power and data representing the product and / or price information to be displayed via the electronic shelf rail 17. A shelf rail controller 18 is used as a central control element, which is also designed for and used for radio reception of the product and / or price information. Such a shelf rail 16, such display signs 17, and the shelf rail controller 18 are known from the international patent applications with the international file numbers PCT / EP2021 / 055914 and PCT / EP2021 / 055916, respectively.

[0100] A video shelf rail 19 is attached to the shelf 5 as a further system component. This video shelf rail also spans approximately half the shelf length and is mounted next to the electronic shelf rail 16. The video shelf rail 19 has a video screen 20 and is designed to display video content as well as still images. It can also be used to display a combination of a background video and one or more superimposed still images. For example, virtual ESLs can be superimposed corresponding to the products positioned on the shelf 5 in order to display the product and / or price information corresponding to the respective product at the appropriate position along the video screen 20. Such a video shelf rail 19 is known, for example, from the international patent application with the international file number PCT / EP2017 / 078845.It has a video shelf rail controller 21, which is designed to wirelessly receive the content to be displayed, represented by display data (video data and / or still image data), and controls the playback of the respective content. An access point 22 is provided for the wireless supply of the two lights 10 and 12, the ESLs 14, the electronic shelf rail 16, the video shelf rail 19, and the camera 13. This access point is connected to the network 3 and converts the wireless data traffic from a wired data transmission to a wireless data transmission, and vice versa. It should be noted at this point that the lights 10 and 12, the camera 13, the electronic shelf rail 16, or the video shelf rail 19 can also be connected to the network 3 via a cable. If power is also required via the network 3 in addition to data, a POE-LAN ​​(Power over Ethernet - Local Area Network) installation can be used for this purpose.In contrast, the individual ESL 14s are typically connected wirelessly to the access point. For wireless power transmission to the system components, a technology known as "Power over Wi-Fi" would be required.

[0101] The ESLs 14 also have an LED 23, which is clearly visible on the front next to their screen 15. It is used for optical signaling and is controlled by the respective ESL controller. The electronic shelf rail 16 also has such an LED 23 for optical signaling, which is also clearly visible on its front and, in this case, is controlled by the shelf rail controller 18. The same can be provided for the video shelf rail 19. However, with the video shelf rail 19, the video screen 20 can also be selectively controlled to emit the optical signal.

[0102] Even though only a very limited number of system components were visualized in the visualization of System 1 chosen here, it should be mentioned for the sake of clarity that several thousand different products may be offered in a retailer's store and therefore the number of system components required for the visualization of product and / or price information as well as marketing messages can be correspondingly high. The same applies to the number of cameras 13, which is roughly based on the number of shelves to be scanned or their length. Typically, smaller groups of these system components are wirelessly combined for communication with a single access point 22, so that the number of access points 22 can be derived from the number of groups. It should also be mentioned that ESLs 14, electronic shelf rails 16, and video shelf rails 19 can be installed in a single installation or as shown in the figure.

[0103] 1 can be used in a mixed installation.

[0104] During operation of System 1, the first software of the server manages

[0105] 2 the ESLs 14, the display signs 17, and the video shelf rail 19 are configured to store the respective spatial position in the store and a logical connection to the respective product in a planogram containing planned position information in digital form, or in a realogram containing actual position information in digital form. The difference between the planogram and the realogram is that the planogram contains planned position information in digital form, whereas the realogram contains actual position information in digital form.

[0106] Knowing this logical link, the individual screen contents to be displayed are transmitted to the respective ESLs 14, to the shelf rail controller 18 for forwarding to the respective display sign 17 installed there, or to the video shelf rail 19. To establish the logical link, the LEDs 23 of the entities of interest, i.e., the LED 23 of one of the ESLs 14 or the LED 23 of the electronic shelf rail 16, on the shelf 5 are first controlled to light up or flash - e.g., by the server 2, which is informed about which product is currently intended to be linked to one of the display devices on the shelf 5 and thus also knows the unique product code. This process can, however, also be initiated locally on the shelf 5, for example, with a Bluetooth or NFC communication between the respective entity 14 or 16, if they are designed for this purpose, and a mobile device (e.g.,a personal digital assistant or a smartphone) of an employee. A picture of the shelf 5 is then taken with the camera 13. The camera 13 now uses the position of the flashing or illuminated LED 23 in the digitalized image of the real scene to localize the respective ESL 14 or the electronic shelf rail 16 as a whole, from which the position of the respective product on the shelf 5 is also determined. The screen content of the respective ESL 14 or of the display signs 17 of the electronic shelf rail 16 is then recognized and evaluated in order to identify the respective entity using a barcode or QR code displayed there. Identification can also be recorded using a visual code (flashing code) conveyed by the LED 23. The position in the digital image and the identity of the respective entity 14 or17 is then communicated by camera 13 to server 2, which is informed of the position of camera 13 and its detection range. From this, the server locates the relevant entity 14 or 17 in the planogram or realogram, respectively, and saves the logical link to the relevant product. Once the logical link has been established and the position of the relevant product on shelf 5 has been determined, the camera 13 can also be used to monitor the development of the inventory on shelf 5 during ongoing operations.

[0107] However, in all these camera-based activities, the presence of a person 7 in the camera's detection area may prove to be disruptive to the further processing of the digital image.

[0108] To counteract this problem, the camera 13 has a radar assembly 24. The radar assembly 24 is shown in block diagram form in Figure 2 and comprises a radar sensor 25 of the modulated continuous wave radar type and a data processing module 26 connected to it. The radar sensor 25 is designed for radar-based object detection within a radar sensor detection range, which is symbolically delimited by the boundary lines 25A and 25B.

[0109] As can be seen from Figure 1, the radar sensor detection area overlaps with the camera detection area. This overlap can be configured to be substantially congruent or—as shown in Figure 1—designed so that the camera can reliably detect the entire shelf 5, whereas the radar sensor 25 can essentially completely detect the shelf 5 and the aisle in front of it.

[0110] As shown in Figure 2, the radar sensor 25, or rather its electronics, is divided into an analog section 27 and a digital section 28. The analog section 27 is used to transmit a radar signal T and to receive a reflection signal R, which is formed by the radar signal T reflected from an object. The digital section 28 is used to process the reflection signal R, whereby, according to a factory configuration of the digital section 28, rudimentary evaluation results AE are provided. However, these evaluation results AE are not used further in this case.Instead of the evaluation result AE, radar sensor raw data RR, provided in real time by the digital section 28, is provided in the form of a digital data stream of the I and Q signals. This data is tapped by the data processing module 26 and continuously analyzed there according to an object classification model or an object prediction model that takes the specific application environment into account. In the present case, the model used is optimized to detect the presence of one or more people in the radar sensor detection area and to indicate their respective position and / or movement state. The result of this analysis is descriptive data BD, which describes in real time the respective radar-detected situation with regard to the presence of one or more people in the radar sensor detection area.

[0111] The integration of the radar assembly 24 into the camera 13 is visualized in block diagram form in Figure 3. In the present case, the electronics 29 of the camera 13, which, in addition to the optoelectronic image acquisition unit 30, has an image evaluation unit 31 with which the previously discussed measures are implemented, is configured such that the camera 13 only performs the image acquisition and evaluation and subsequently transmits evaluation data AD representing the evaluation result to the server 2 via its radio interface if the description data BD indicate that the radar sensor detection area is free of persons.This allows, on the one hand, the energy consumption of camera 13 to be significantly reduced, the battery life to be increased, and, at the same time, the radio channel usage for transmitting the evaluation data AD to the necessary level, because the evaluation data AD only occurs and needs to be transmitted when the captured image of the scene in which shelf 5 is present could actually be captured without interference from people. For periods in which the radar sensor detection area is not free of people, camera 13 could not only be prevented from capturing and processing data, but could also be switched to a standby mode with significantly reduced power consumption. However, camera 13 can also be configured to continuously transmit the description data BD of radar module 24 to server 2 via its radio interface. This allows server 2 to optionally use the description data BD originating from camera 13.Thus, whenever the description data BD indicate that the relevant aisle section in front of the shelf 5 is free of people, the server 2 can dim or deactivate the light 10 in order to reduce its power consumption, and only activate the usual brightness as soon as the description data indicate that the relevant aisle section in front of the shelf 5 is not free of people.

[0112] This measure, discussed using a single camera 13, can also be implemented using all cameras installed along the entire corridor (not visualized) to coordinately control the ensemble of lights 10 installed along the entire corridor via the server 2. Thus, the server 2 can operate the lights 10 located near it with increased brightness, corresponding to the respective actual position of the person 7 along the corridor, while lights 10 positioned further away from the person 7 are operated with reduced brightness, continuously adjusting these distortions to the actual position of the person 7.

[0113] The first luminaire 10 can also have its own radar module 24 (not visualized), whose description data BD is used locally, as previously discussed in connection with the camera 13. In this case, it is used by a luminaire controller (not shown) integrated into the luminaire 10 to control the brightness and ultimately to optimize the power consumption of the luminaire 10 for energy savings. The description data BD generated by the luminaire 10 can also be continuously transmitted to the server 2 for further processing purposes.

[0114] The aspects discussed in connection with the first luminaire 10 also apply to the second luminaire 12.

[0115] The video shelf rail 19 shown in Figure 1 also includes the radar module 24. Here, the description data BD is transmitted to the video shelf rail controller 21, and the playback of static or dynamic content is adjusted according to the information content of the description data. The overall block diagram structure of the video shelf rail corresponds to that shown in Figure 3, although in this case, the electronics 29—in this case, the electronics 29 of the video shelf rail 19—is also connected to the radar module 24.

[0116] In this specific case, the video shelf rail 19, when the radar sensor detection area is free of people, visualizes a static background image and, corresponding to the respective product position along the video shelf rail 19, a virtual ESL with the relevant product and / or price information. This reduces the power consumption of the video shelf rail 19 compared to the playback of dynamic content such as a video. As soon as the description data BD indicates the presence of a person, the shelf rail controller 21 changes the displayed content by leaving the virtual ESLs as they were previously, but instead of the static background image, plays a background video intended to attract the attention of person 7. As soon as person 7 leaves the radar sensor detection area, the system switches back to the static background image.Analogous to the functionality of camera 13, in the case of the video shelf rail 19, a standby mode of the video shelf rail 19 could also be activated when a person-free radar sensor detection area is displayed, or at least the backlight of the video screen 20 could be dimmed to realize further energy savings potential. The descriptive data BD generated from the video shelf rail 19 can also be continuously transmitted to the server 2 for further processing purposes.

[0117] The ESLs 14 can also have a radar assembly 24, as is the case with the ESL 14 attached to the shopping cart 6. In this case, however, the radar assembly 24 does not directly influence the operating behavior of the ESL 14, but rather purely records the activities in the respective radar sensor detection area. Thus, the description data BD is always immediately transmitted to the server 2 to make it available there for further processing purposes.

[0118] However, the radar assembly 24 can also be present as an individual radar assembly device 32, i.e., without integration into another, higher-level device. Such a radar assembly device 32 is mounted in the center of the top shelf of the shelf 5 and serves to independently detect persons 7 in front of the shelf 5. In contrast to the radar assembly 24, the radar assembly device 32 has an additional module radio module 33, which is shown in Figure 2 with a dot-dash line and bears an antenna symbol, for communication with the access point 22. This module radio module 33 can be integrated into the data processing module 26 or formed separately and connected to the data processing module 26. The radar assembly device 32 is also shown in Figure 2 with a dot-dash line.The housing of the radar assembly device 32 is preferably designed to correspond to the structure to which it is to be attached, placed or hung.

[0119] The description data BD as such represents the respective current situation in the respective radar sensor detection range. They represent a snapshot of this situation at a given time by the respective radar sensor and, from a functional perspective, describe the respective situation as a function of the respective detection time. This system is visualized using Figure 4 for a single radar sensor assembly 24, where a sequence of description data BD is depicted along the time axis t in the form of data records DS1 ... DS15, each corresponding to one of the detection times t1 ... t15, for this one radar sensor assembly 24. The data records DS1 ...DS 15 may have a radar sensor identifier or a radar assembly identifier in order to uniquely assign it to a specific radar sensor 25 or the radar assembly 24 in question, wherein the position in the business premises or the assignment to another device in which the radar assembly is integrated and of which the position is known is stored for the radar sensor or the radar assembly in question, e.g. in the planogram or the realogram or a further database linked thereto and is therefore known.

[0120] In order to make the basic properties of the description data BD accessible for further use, according to a preferred embodiment, the description data BD of all radar modules 24, whether they are integrated into a device with a higher-level functionality or are a component of the individual radar module device 32, are provided via the network 3 and digitally stored. For example, the local server 2 can use the description data BD for further evaluations via appropriately programmed software routines, for example, to determine dwell times, movement patterns, and clusters of the persons 7, etc., and to draw conclusions therefrom.Preferably, however, the description data BD are transmitted directly to a cloud-based provisioning software 34, symbolized in cloud 4 in Figure 1, which allows random access from the retailer's software system or a back-up retailer. Furthermore, the cloud-based provisioning software 34 also allows access by a third-party software system in accordance with the access rights granted to the third party by the retailer or retail chain.

[0121] The provisioning software 34, whose functional structure is visualized in Figure 5 and which in the present case is implemented with the aid of a programming interface, also known as an application interface, often abbreviated to API 34 for "application programming interface", essentially has two functions, namely, on the one hand, a storage function 35 for the description data BD and, on the other hand, a provisioning function 36. This enables access to the individual current description data BD of all radar modules 24 as well as their historical description data BD. In addition, the API 34 is programmed in such a way that it can generate a summary, represented by summary data ZD, of the available description data BD. The manner in which this summary is generated depends on a parameterization P, which the API 34 receives from a querying instance 37. The type of parameterization P, such asA function call and the arguments to be passed, etc., are predefined for the API 34 and known to the calling instance 37. To obtain relevant results, the parameterization P specifies at least the radar sensor or radar module in question, if applicable, the time range of interest, and the type of summary. By way of example and by no means exhaustive, it should be noted that this parameterization P:

[0122] - to create a targeted data set extract, such as in the present case the data sets DS3, DS6, DS9, DS12 and DS15, in periodic or otherwise specified (possibly recurring) time periods, which is visualized in Figure 4 with the help of first summary data ZD1.

[0123] - to determine the number of persons N counted in a time period, such as between t2 and t5 as well as between t11 and t15, in the radar sensor detection area in question, which is visualized in Figure 4 with the help of second summary data ZD2.

[0124] - to determine the average duration of stay T of the persons present in the radar sensor detection area in question, which is visualized in Figure 4 using third summary data ZD3.

[0125] - to determine the standstill time, i.e. the dwell time TR in the rest state, of one or more detected persons in the radar sensor detection area, which is visualized in Figure 4 with the help of the fourth summary data ZD4.

[0126] - to determine the time periods TE during which the radar sensor detection area in question was free of people, which is visualized in Figure 4 using fifth summary data ZD5. etc.

[0127] For clarification, with regard to Figure 4, the summary data ZD1 ... ZD5 entered along the time axis t only visualize the data sets DS1 ... DS15 to which they refer or the temporal context to which they refer. The summary data ZD1 ... ZD4 can be generated independently of this time axis t, i.e., typically after the data sets DS1 ... DS15 have been generated and saved.

[0128] The query instance 37 can, for example, be lighting control software 38 (see Figure 6) that is processed on the local server and with the help of which the luminaires 10 and 12 are controlled in a centrally managed manner at the respective retailer or in their store. This allows, for example, ad hoc decisions to be made about the required brightness in the vicinity of the respective luminaire by using the current description data BD and to implement these decisions in real time when a person is detected or the radar sensor detection area is free of people. In addition, lighting profiles can be generated from the historical description data that predefine the lighting requirement, for example depending on the time of day, day of the week, or public holiday, etc., so that based on these lighting profiles, the actual current required lighting requirement is adapted and triggered via the current description data.The lighting profiles can also be continuously adapted over time to incorporate global changes in customer behavior. These measures make a significant contribution to energy savings in terms of lighting in the retail space, not only in the short term (momentarily) but also in the long term. Controlling the light intensity can also affect the backlighting of the video shelf rail 19. This allows for an adaptive power consumption management process across a wide variety of electronic devices, ensuring that power consumption is only activated or increased when it actually appears necessary.

[0129] The querying instance 37 can also be content management software 39 (see Figure 6), which is used to allocate and control the content to be played back using the video shelf rail 19. This allows, for example, ad hoc decisions to be made in real time using the current description data BD regarding the video and / or still image content that best suits the respective situation in the radar sensor detection range. Furthermore, playback profiles can be generated from the historical description data BD, which can define the playback content, for example, depending on the time of day (e.g., a breakfast video is played in the morning hours), or depending on the day of the week (e.g., a party video is played from 4 p.m. on Friday), or even depending on a public holiday (a Christmas video is played on the days before Christmas), etc.predefine so that, based on these playback profiles, which define the dominant long-term content for the respective time period, the currently required content is played back triggered by the current description data BD. For example, in the morning hours, a steaming cup of coffee can be zoomed into the foreground when a person is detected. For example, on Friday from 4 p.m. onwards, a packet of chips can be zoomed into the foreground when a person is detected. For example, in the days before Christmas, a notice about reduced-price gingerbread can be displayed when a person is detected, and so on. These playback profiles can also be adapted over time to changing customer behavior. A process for the dynamic placement of advertising or marketing messages can therefore be implemented, which ensures that the advertising orthe marketing messages are placed precisely when they actually have the potential to attract a person's attention and subsequently trigger a purchase impulse.

[0130] The querying instance 37 can also be a camera control software 40 (see Figure 6), which is responsible for controlling the operating behavior of the camera 13. This allows, for example, ad hoc decisions regarding image capture by the camera 13 or the use of the detection data supplied by the camera 13 to be made and implemented in real time by using the current description data BD when a person is detected or the radar sensor detection area is free of people. In addition, activity profiles can be generated from the historical description data BD, which indicate at which times or in which time periods (e.g.of a day) the detection activity of camera 13 should preferably take place or the data received from camera 13 should be analyzed preferentially because, based on the historical description data BD, a person-free radar sensor detection area is predicted with high probability for these time periods. The activity profiles can predefine the activity of camera 13 or the further processing of the data supplied by camera 13, e.g., depending on the time of day, weekday, or holiday, etc. In this scenario, too, the current description data BD can be used to deviate from the respective applicable activity profile depending on the occasion.In the context of the detection of the shelf 5 with the aid of the camera 13, an ESL control software 41 (see Figure 5) can also be used as the querying instance 37, which controls the functionalities of the ESLs 14 and triggers the lighting or flashing of the LED 23 on the ESL 14 that is intended for establishing the link with a product if the camera detection area is described as free of people, in order on the one hand to keep the energy consumption of the battery-operated ESL 14 low and on the other hand to focus the image detection and evaluation activity of the camera 13 on the relevant ESL 14 or so that a downstream evaluation software using the image data of the camera 13 can focus on this lighting or flashing.A process can therefore be implemented to optimize energy consumption through image-based automatic detection of shelf status, as well as through the automatic image-based creation of a logical link between a product and your ESL 14. This goes hand in hand with optimized resource utilization in terms of data processing power and radio channel usage.

[0131] The querying instance 37 can also be customer flow analysis software 42 (see Figure 6) that analyzes the customer flows in the store based on the description data BD of one or more radar assemblies 24. This analysis can be limited to the description data BD of a single radar assembly 24, whereby the number of people N, their average length of stay T or their dwell time TR in front of a single shelf 5 or a single shopping cart 6, etc., will essentially be of interest in order to make statements about interest or disinterest in the product in question and ultimately the brand behind it, which are assigned to the relevant radar sensor detection area. This allows for the creation of a dynamic method for recording the product or brand perception of customers in the store, which reflects the actual situation in the entire store at any given time.Retrospectively, product or brand awareness statistics can be derived from this.

[0132] Furthermore, it is advantageously provided that an event corresponding to the respective summary data ZD1 ... ZD5, which event is described by event data respecting the event, is recorded together with the relevant summary of the description data BD, stored and also made available via the API 34 for later evaluations and optimizations of the behavior or operation of the system 1. Such an event can be given by an internal event, such as the settings of the various devices for controlling or influencing the power consumption in the system 1 or the played videos or the set playlists or the selection of certain time ranges, etc. However, such an event can also be given by an external event (such asA specific event may be present (e.g., a calendar day, a public holiday, a vacation period, the average daily temperature, the maximum daily temperature, the weather during business hours, a catastrophic event in a region in which the business premises in question are located, etc.). The respective event is recorded digitally, i.e., either internally in System 1 or via an internet connection, and stored, in particular, assigned to the respective summary data ZD1 ... ZD5, for example.

[0133] Furthermore, it is advantageously provided that the respective summary data ZD 1... ZD5 are supplemented by supplementary data ED which originate from a data source other than the respective radar sensor 25 or the radar assembly 24.

[0134] These supplementary data ED can be obtained, for example, from a retailer's smartphone application (not shown), with which a person 7 (a customer) standing in front of the shelf 5 scans a QR code displayed on the screen of one of the ESLs 14 or carries out an NFC interaction with the relevant ESL 14, which must be NFC-capable for this case, after which, for example, a web link is called up or a discount coupon is activated, etc. The same applies mutatis mutandis to the electronic shelf rail 16 or the video shelf rail 19. Such supplementary data ED represent interaction actions directly on the shelf 5 with a direct reference to the respective product to which the ESL 14 or the electronic shelf rail 16 or the video shelf rail 19 is assigned.

[0135] The supplementary data ED can also be obtained automatically from a cash register system or a similar device, such as an inventory management system, where, for example, the number of products sold or the product-related turnover is saved. Such supplementary data ED represents a sales success. All of this supplementary data ED can be related to the time period of interest, which also forms the basis of the respective summary data ZD1 ... ZD5. This supplementary data ED can also be recorded and made available via the API 34, for example, as part of the respective summary data ZD1 ... ZD5 in accordance with the parameterization of the querying instance 37. The querying instance 37 can be a marketing and success analysis software 43 (see Figure 6), which is either used internally by the retailer or externally by a marketing or advertising agency or by a supplier orProducers of a product. Only with the help of the recording data ED can the conversion rate be calculated for the relevant interested parties, which indicates the success of the respective pricing, marketing and / or advertising measure, whereby this measure was preferably implemented by appropriate control of the affected ESL 14, the electronic shelf rail 16 or the video shelf rail 19. Only in this way is an objective evaluation of this measure (generally referred to as a campaign) possible, because the customer behavior in the business area (i.e. QR code scanning or NFC interaction) is available directly in real time via the current description data BD or the customer behavior present during an ongoing measure after completion of the respective measure, summarized by the respective summary of the description data BD, supplemented by the interactions that took place as well as the sales success.Building on this, a success-based billing or pricing model can be established that is not based on a fictitious success rate established by surveys and the like, but rather on facts represented by the collected supplementary data ED.

[0136] Even if the supplementary volumes ED are not accessible or are not made accessible, it is possible to establish a location-dependent billing or pricing model using only the summary of the descriptive data BD. Since each product, product group, or brand can be assigned a radar sensor 25 or a radar module 24, which radar-based captures the movement area of ​​people in front of a product presentation area—i.e., the respective shelf 5, the respective product counter, the respective display case, or the respective shopping cart 6—the summary of the descriptive data BD can be used to determine objective customer behavior in front of the respective product presentation area.This makes it possible to precisely determine whether and for what average length of stay (T), in particular the dwell time TR, customers are in front of the respective product presentation area, which is a clear measure of the attention a person (7) pays to the product presented there or to the related marketing or advertising measure presented there. The entire customer flow present there can thus be categorized into potentially interested people who stop in front of the product and spend time there and therefore must be counted, and potentially uninterested people who move quickly past or only briefly stay within the radar sensor detection range and therefore may / should not be counted.All of this information can be retrieved fully automatically from the API by the querying instance 37 using suitable parameterization P, and can subsequently be used for the pricing model for pricing the respective merchandise location for the brand manufacturers. Ultimately, the transparent availability of the description data BD, as well as their essentially random summary and provision of the summary data ZD1 ... ZD5 etc., even allows the establishment of a pay-per-view payment model for specific merchandise presentation locations in the store. In this case, the respective dwell time TR, in particular a threshold value defined for this purpose using the parameterization P, can be taken into account, so that a fee must be paid for persons who stay in front of the merchandise presentation location longer than defined by the threshold, because for such long dwell times it can be assumed that the product or service is available.The marketing message conveyed there has definitely been perceived and there is significant interest in it. This information (as well as the applied parameterization P down to the underlying descriptive data BD) is now no longer only accessible to the retailer, but can also be made completely transparent to the brand producer or the marketing or advertising agency responsible for the brand via API 34. This ensures, first and foremost, that the collaboration between the retailer and the producer or brand manufacturer is completely transparent when charging product presentation prices.

[0137] Figure 6 is discussed below. It illustrates, by way of example, the central function of the API 34 and the aforementioned accessing instances 37, namely the lighting control software 38, the content management software 39, the camera control software 40, the ESL control software 41, the customer flow analysis software 42, and the marketing and success analysis software 43, which either access the current description data BD in real time or, using individual parameterization, query a summary of the description data BD that is relevant to the respective accessing instance 37 ... 43. The querying instance 37 evaluates the current description data BD or the summary of the description data BD and executes a function SF, as discussed with reference to instances 37 ... 43.

[0138] The following, with the help of Figure 7, describes the basic method 44, which is carried out using the described system 1. The method starts at a block 45, in which the radar sensors 25 are assigned to locations in a store that can be radar-detected using their individual radar sensor detection range. These locations can be an area in front of the shelf 5, in front of a presentation table, in front of a display case, in front of the shopping cart 6, in front of a camera 13, etc.

[0139] According to a further block 46, the radar sensors 25 continuously detect the respective situation in the respective radar sensor detection range from their respective position and continuously transmit the radar raw data RR generated thereby.

[0140] According to a further block 47, these radar raw data RR are continuously analyzed within the radar assembly 24 using the data processing module 26 connected to the radar sensor 25, according to the object classification model and / or object prediction model used there. These software- and / or hardware-implemented models are used to determine whether at least one person 7 is present within the radar sensor's detection range and, if presence is detected, to also determine the location within the radar sensor's detection range.

[0141] According to a further block 48, as a result of applying the object classification model and / or object prediction model to the radar raw data RR, the description data BD is generated by the data processing module 26, whereby the situation present in the respective radar sensor detection area is described, i.e. essentially a person-free detection area or a detection area with a person or persons present.

[0142] According to a further block 49, the description data BD as such is forwarded to the cloud-based programming interface, i.e., to the API 34, in order to make the respective snapshot of the situation in the respective radar sensor detection range of the radar sensors 25 or the radar modules 24 available there with maximum granularity for the querying instance 37. However, the API 34 also provides the summary of the description data BD according to the parameterization P defined by the querying instance 37 and thus allows a random evaluation of the respective description data BD of the radar sensors 25 or radar modules 24 selected according to the parameterization P or the access rights of the querying instance 37. The API 34 processes query requests occurring in parallel from different querying instances 37, which can originate either from within the system 1 or from outside the system 1.The admissibility of a query (access to specific radar modules / the type of parameterization, etc.) can be regulated by privileges that can be individually defined for different querying instances 37. In particular, the API 34 can distinguish between system-internal and system-external queries and accordingly allow different parameterizations and thus different evaluations or summaries for system-internal and system-external queries.

[0143] Blocks 46 to 49 are de facto executed continuously during the execution of the method, because parallel processes also occur, which, however, cannot be visualized. The method ends in block 50, e.g., by deactivating radar acquisition or by terminating the processing of the radar raw data RR, but in any case by terminating the provision of the description data BD or the summary of the description data BD by API 34.

[0144] In summary, the measures discussed ensure that the respective description data BD of the individual radar sensors 25 are centrally available via API 34 for all authorized querying instances 37 and can also be centrally summarized there according to the respective parameterization. This allows, particularly within the system, the operating behavior of the system components located there to be automatically adapted depending on the query results from API 34. API 37 plays a central role here because it provides the description data and / or its summary for each radar sensor individually for use within the retail system and / or - depending on the applicable access rights - for use outside the retail system.

[0145] Finally, it should be noted once again that the figures described in detail above are merely exemplary embodiments that can be modified in a variety of ways by those skilled in the art without departing from the scope of the invention. For the sake of completeness, it should also be noted that the use of the indefinite articles "a" or "an" does not exclude the possibility that the relevant features may be present multiple times.

Claims

Claims 1. A computer-assisted method (44) for processing radar data in a radar-assisted retail system (1), the method (44) comprising the following method steps, namely: - a continuous output (46) of radar raw data (RR) from one radar sensor (25) of a plurality of radar sensors installed in a sales area, wherein the radar raw data (RR) represent a radar-detected situation in a respective radar sensor detection range of the respective radar sensor (25), - a continuous analysis (47) of the radar raw data (RR) for the purpose of generating description data (BD), which description data (BD) digitally describe the respective detected situation with regard to: at least the presence of a person (7) in the radar sensor detection area, preferably also with regard to the location of the person (7) in the radar sensor detection area if presence is detected, - providing (49) the description data (BD) as such and / or a summary of the description data (BD) via a programming interface.

2. Method (44) according to claim 1, wherein - the radar sensor (25) is designed as a frequency modulated continuous wave radar, abbreviated to FMCW radar, and - the modulated continuous wave radar as the radar raw data (RR) a digital representation of its received signal comprising an in-phase component, which is referred to in English as "in-phase component signal", abbreviated as I-signal, and a reactive component, which is referred to in English as "quadrature component signal", abbreviated as Q-signal, and - the ongoing analysis is carried out directly on the basis of the in-phase component and the reactive component.

3. Method (44) according to one of the preceding claims, wherein the continuous analysis of the radar raw data is carried out with the aid of a data processing module (26) which is connected to the respective radar sensor (25).

4. Method (44) according to claim 3, wherein the respective radar sensor (25) and the data processing module (26) connected thereto form a radar assembly (24), in particular a radar assembly (24) integrated into a system component (10, 12, 13, 14, 16, 19) of the retail system.

5. The method (44) according to claim 3, wherein the data processing module (26) applies at least one object classification model and / or at least one object prediction model during the ongoing analysis.

6. The method (44) according to claim 5, wherein the data processing module (26) has a machine learning mode and in the machine learning mode the radar raw data (RR) are used to create the at least one object classification model and / or the at least one object prediction model.

7. Method (44) according to one of the preceding claims, wherein the programming interface (34) is contacted by a querying entity (37) and, depending on the description data (BD) available according to the query via the programming interface (34) or their summary - a behavior of a system component (10, 12, 13, 14, 16, 19) of the retail system is controlled and / or - data provided by the system component (13, 14, 16, 19) are interpreted.

8. The method (44) according to claim 7, wherein - the radar raw data (RR) are identified by a radar sensor identifier which allows a clear identification of the radar sensor (25) from which the radar raw data (RR) originate, and - the description data (BD) are identified by the radar sensor identification.

9. Method (44) according to claim 7, wherein the system component is formed by a camera (13) which is designed to deliver a digital recording of a scene in its camera detection area, and wherein with the aid of the querying instance (37), if the description data (BD) or their summary indicate that a radar sensor detection area which also covers the camera detection area is free of persons, - the camera (13) is at least temporarily caused to record the scene or - the digital recording of the scene provided by the camera (13) is marked as person-free.

10. The method (44) according to claim 7, wherein the system component is formed by a display device (14, 16, 19) which is located within the camera detection range, and wherein with the aid of the interrogating instance (37) under the condition of the person-free radar sensor detection range, the display device (14, 16, 19) is caused to emit a light signal which can be detected with the aid of the camera (13).

11. Method (44) according to claim 7, wherein the system component is formed by a lighting device (10, 12, 20) which is adapted to be adapted with regard to its light output << intensity / colour / signalling ... >> and which is designed to illuminate a Radar sensor detection area and / or an object present in the radar sensor detection area or for emitting light from an object located in the radar sensor detection area, and wherein with the aid of the querying instance (37), if the description data (BD) or their summary indicate a person-free radar sensor detection area, - the light emission from the lighting device (10, 12, 20) is prevented or - the light output caused by the lighting device (10, 12, 20) is changed at least in comparison to a light output which is set when a person (7) is present in the radar sensor detection area.

12. Method (44) according to claim 7, wherein the system component is formed by a display device (19) which is designed to display a content perceivable by a human within a perception range, and wherein with the aid of the querying instance (37), - if the description data (BD) or its summary indicates a person-free radar sensor detection area covering the perception area, the display device is caused to display static content, and, - if the description data (BD) or their summary indicate the presence of a person in the radar sensor detection area, the display device (19) is caused to display dynamic content, or vice versa.

13. Method (44) according to claim 7, wherein the system component is formed by a sensor different from the radar sensor, which detects a physical parameter such as humidity, brightness or temperature, etc., and wherein with the aid of the querying instance (37), if the description data (BD) or their summary indicate that in a person is present in the radar sensor detection area, - the detection of the physical parameter by the sensor is at least temporarily suspended or - Detection data representing the detected physical parameter and emitted by the sensor are marked as having been recorded in the presence of a person.

14. The method (44) according to claim 1, wherein the programming interface (34) generates the summary of the description data (BD) depending on a parameterization (P) transmitted to it.

15. The method (44) according to claim 14, wherein the parameterization (P) specifies at least one of the radar sensors (25) for which the summary of the description data (BD) is to be generated according to the parameterization (P), and the programming interface (34) applies the parameterization (P) to the description data (BD) generated from the radar raw data (RR) of the specified radar sensor (25).

16. The method (44) according to claim 14, wherein - the parameterisation (P) specifies a temporal aspect and / or a quantitative aspect of the radar-detected situation, and - the programming interface (34) summarizes and provides a sequence of the description data (BD) generated from the radar raw data (RR) of at least one of the radar sensors (25) with regard to the specified temporal aspect and / or with regard to the specified quantitative aspect.

17. The method (44) of claim 14, wherein the summary of the description data (BD) is provided together with event data describing an event generated by the retail system or an external event whose occurrence corresponds to the summary of the description data (BD).

18. Method (44) according to claim 14, wherein the summary of the description data (BD) is supplemented by supplementary data (ED), wherein the supplementary data (ED) are obtained from a data source other than one of the radar sensors (24).

19. Method (44) according to claim 18, wherein the supplementary data (ED) represent at least one of the following information: - the number of QR code scans, - the number of web link views, - the number of NFC interactions, - the number of units of a product sold - the total sales volume of a product sold.

20. The method (44) according to claim 18, wherein the supplementary data (ED) are generated for a period of time given by the parameterization (P).

21. The method (44) according to claim 19, wherein the summary of the description data (BD) is provided together with the supplementary data (ED).

22. Method (44) according to claim 1, wherein the programming interface (34) processes a plurality of parallel accesses by different software applications, in particular by different querying instances (37).

23. The method (44) according to claim 1, wherein the programming interface (34) itself is hosted in a cloud (4).

25. Radar-assisted retail system (1) adapted to carry out the method (44) according to any one of the preceding claims.

Images