How to create a digital image of your sales area

Abstract

A method for producing a digital image of a sales area, the method comprising the steps of: generating a digital image of the sales area by computer processing using recordings from at least one variably positioned camera device providing a digital image of the surroundings of a path of travel within the sales area; and updating a portion of the digital image of the sales area by computer processing using recordings from at least one fixedly positioned camera device providing a digital image of a portion of the sales area.

Description

[Technical Field]

[0001] The present invention relates to a method for creating a digital image of a sales area. [Background technology]

[0002] For example, when digitizing a sales area such as a supermarket sales floor, autonomous robots are used today that move through the supermarket and use cameras to take photoelectric digital images of the supermarket's interior equipment, such as racks, shopping baskets, and possibly the products displayed therein.

[0003] The digital images produced are of very high quality and virtually free of optical distortion, and are therefore extremely suitable for digitizing sales areas, in the sense that they produce a "rearogram" that represents the actual appearance of the sales space, along with all the fixtures and merchandise displayed therein.

[0004] However, in terms of cost, the investment to acquire such robots is high. Likewise, the operation of such robots entails high operational costs. Furthermore, the presence of one or more such robots in a sales area often disturbs, makes customers feel uneasy, or even feels like a nuisance. This means that, when used during business hours, such robots must generally be operated very slowly due to safety considerations, which means that full digitalization of a sales area is often only possible once a day, or even less frequently. Due to the high cost, the use of a single robot is usually preferred over the option of using multiple robots simultaneously. The use of such robots outside of business hours does not significantly change the aforementioned situation, since it must be taken into account that personnel, such as supermarket staff or external service personnel, are present in the sales area even during non-business hours.

[0005] In any case, the digitization of the sales area created using such robots lacks real-timeness, for the reasons mentioned above, since the update cycle is too long to be effective during business hours, and updates outside business hours in any case only provide an uncertain contribution to the real-timeness of the situation within the sales area. Summary of the Invention [Problem to be solved by the invention]

[0006] In view of this background, it is an object of the present invention to provide a method that overcomes the aforementioned problems. [Means for solving the problem]

[0007] This problem is solved by the method according to claim 1. The subject of the invention is therefore a method for creating a digital image of a sales area, comprising the following steps of computer-generated generation of the digital image of the sales area using at least one variably positioned camera device providing a digital image of the surroundings of a movement path within the sales area, and computer-generated updating of a part of the digital image of the sales area using at least one fixedly positioned camera device providing a digital image of a part of the sales area.

[0008] The measures according to the invention have the advantage that, using the method or a system configured and / or programmed to implement the method, a digital image of the sales area can be realized that can be kept up-to-date for at least one portion of the sales area captured by a fixed camera or for multiple portions of the sales area captured by fixed cameras. A "wide" digital image of the actual sales space, which usually has a relatively large surface area and a high richness of three-dimensional detail, realized using one or multiple variable-position camera devices is instantly updated using the fixed-position camera(s). This introduces dynamism or real-timeness, along with a depth of detail that cannot be achieved with one or more variable-position camera devices, to a digital representation of the wide area, and in particular the entire sales area, which typically appears "static" (due to the relatively slow capture method of the variable-position camera devices) because the variable-position camera devices cannot always guarantee to capture the parts of the sales area that truly require real-time capture, for example, due to product shortages, queues, etc.

[0009] Unlike variable position camera equipment, fixed position camera equipment certainly only images a predetermined portion of the sales area, where the reference position does not change and so is always available to image that portion of the sales area.

[0010] Based on this basic finding, by suitable extension of this basic measure, i.e. by suitable number, suitable positioning and suitable orientation of fixed-position camera devices, real-timeness can be guaranteed for the interesting part of the sales area, or even for the entire sales area. The resulting digital image of the sales area can also be interpreted as a "living digital twin" of the actual sales area, since this digital twin truly embodies real-timeness.

[0011] Further particularly advantageous embodiments and refinements of the invention emerge from the dependent claims and the following description.

[0012] A sales area can be, for example, the sales space of a supermarket, equipped with all the necessary equipment, such as racks, baskets, tables, and sales areas for displaying products. Similarly, a sales area can be, for example, the storage area of ​​a building materials retail store or a logistics center, where, in addition to the aforementioned racks, baskets, tables, and the like for displaying and storing products, heavy-duty racks are also installed in the sales area, storage area, or outdoor area. In this case, the term "sales area" is not limited to the aforementioned supermarkets and building materials retail stores, but can encompass various types of retail and wholesale businesses. The term "sales area" can also encompass a combination of indoor and outdoor areas, such as the checkout area where customers wait, possibly in line, to pay for their selected products, and other areas not directly reserved for the sale of products, such as customer service centers and parking lots.

[0013] A positionable camera device can be configured to capture scenes within its field of view in the form of still image records or video recordings. The device generates digital still image records or sequences thereof, or even digital video recordings. It can also be provided, for example, that the camera device generates both still image records and video recordings, depending on the requirements, where the still image records and video recordings are generated time-shifted relative to each other, or the still image records can be generated during video recording (e.g., separate from ongoing video recording). For this purpose, the camera device is equipped with means typically provided for this purpose, such as a lens and an image sensor, whereby an object or scene within the camera's field of view is projected or projected through the lens onto the image sensor. Based on this projection, the (typically semiconductor-based) image sensor generates a digital image of the object or scene within the camera's field of view. The camera electronics, which include at least one programmable processor and peripheral electronics providing the camera device's functionality, further process the generated digital images. The digital images of the object or scene thus obtained may be temporarily stored in digital form in the camera device and / or provided in a data-reduced form through data compression, and then transmitted to a data processing device using a wireless stage, such as an IoT (Internet of Things) wireless stage using 4G, 5G (the "fifth generation technology standard for broadband cellular networks"), or successor wireless standards. For this purpose, other wireless technologies, such as the widespread WLAN standard, such as IEEE 802.11g, IEEE 802.11h, IEEE 802.11n (also known as WiFi4), or successor standards, may also be used. The same applies in a similar sense to fixed-position camera devices.

[0014] The data processing device may consist of a local computer or server of the sales area operator, a computer center allocated to the sales area operator and reachable via the Internet, or a cloud-based software solution hosted in a computer center (server farm) and available to the sales area operator via the Internet. Software runs on the data processing device. This software is provided with the recordings from the variable-position camera devices and the fixed-position camera devices in digital form. This software processes these recordings using the functions described below.

[0015] In a first aspect, the generation of a digital image of the sales area involves processing the surroundings of the path of movement within the sales area, photographed using at least one positionable camera device, using image processing software, in particular using software means known in the technical term as computer vision, and as a result of this processing, a virtual space is realized as a digital image of the sales space, which virtual space is made accessible in a form of visualization that can be perceived by a person using visualization equipment, in particular a screen or an equipment known in English as a "Virtual-Reality-Headset".

[0016] Specifically, to create this virtual space, image processing software can be used which detects objects (rack aisles, racks, baskets, tables, etc.) in a series of recordings along the path of movement, here being a sequence of still image recordings or video recordings, and describes these objects in digital form and aligns them with each other, thus realizing a digital representation of the sales area.

[0017] To support this process, reference points can also be provided within the sales area that are known to the image processing software (because they are pre-stored therein). These reference points can be radio beacons (known in the technical term "radio beacons") that transmit signals that are received, for example, by the positionable camera and used to measure its position relative to the radio beacon. This information is provided to the image processing software, allowing for more precise positioning of the recordings made with the positionable camera equipment in relation to the actual sales area. However, these reference points can also be optically perceptible reference elements or reference signals whose position within the sales area is known or that identify unambiguous positional information within the sales space. They can also be simple geometric shapes. They can also be coded information carriers, for example in the form of a QR code (for "Quick Response" in English), an infrared light signal, or a laser light signal. These optically perceptible reference elements are included in the recordings of the positionable camera equipment and can be identified there (and, in some cases, decoded if they contain coded information). The meaning is known to the image processing software, since they are either pre-stored or pre-programmed with a method for processing visually perceptible reference elements or reference signals that can be used, on the one hand, to define the position of the positionable camera device and, on the other hand, to position the recordings made with the positionable camera device within the digital image of the sales area.

[0018] However, the construction of a virtual space can also be carried out completely autonomously, i.e., without the use of reference points within the sales area, using image processing software. In this context, the use of software tools, technically known as "computer vision" ("Computer Sehen" in German and its synonyms in English), has proven particularly advantageous. The field of computer vision is well known at the time of filing this application. The software tools used there process and analyze images captured by a camera in various ways to understand their content or extract geometric information, which then allows the creation of a virtual space as a "digital twin" of the actual sales area. A "digital twin" is a data structure that stores geometric information or objects detected using CV, as well as their spatial relationships with each other. In this way, for example, the aforementioned furniture in a store can be easily detected, and the paths between them, their extensions, and their orientations with respect to each other can also be measured. For each detected object or structure, a data structure, e.g., vectors, can be stored that also define the vertices and intermediate points that define the boundaries of the object or structure. These vectors can be associated with the origin and / or neighboring objects or structures. To ensure that the detected geometric information thus obtained, and ultimately the 3D polygons (or polygon meshes) describing the detected objects, can be reproduced in a highly detectable and human-perceptible form, the 3D polygons can be textured. This texturing of the 3D polygons can be performed, for example, using recordings from a variable-position camera and / or a fixed-position camera to create a realistic impression. Alternatively, texturing can be performed using predefined image elements to obtain a more abstract representation, possibly with less precise details.

[0019] In the following, where appropriate, for reasons of brevity, expressions referring to software means of computer vision will be abbreviated to "computer vision".

[0020] To allow a person to access the spatial information of objects in the sales area stored in the data structure, visualization software accesses the data structure and drives visualization devices, such as screens and virtual reality headsets, according to the user's desired view of the sales space. This visualization software runs locally on the data processing device and its control and / or content data can be transmitted to each visualization device, for example by wired connection in the case of a screen or wirelessly, and advantageously wirelessly in the case of a virtual reality headset, for example using the wireless technology arrangements mentioned above.

[0021] In yet another aspect, updating the digital image of the sales area includes processing the portion of the sales area photographed using at least one fixed-position camera device by image processing software, in particular by software means technically referred to as "computer vision", wherein portions of the digital image of the sales area generated using the variable-position camera devices that correspond to the portion of the sales area photographed using the fixed-position camera devices are modified in whole or in detail based on the recordings of the fixed-position camera devices.

[0022] The above statements also apply to this step of the method, with regard to the image processing software or computer vision software means used, where they apply only to the part of the sales area that is within the field of view of the fixed-position camera equipment, so that both events and changes in this part of the recording area can be incorporated into the data structure in a precisely targeted manner, with any desired frequency, and even in virtually real time, so that the "digital twin" of the sales area has the desired real-time characteristics, in particular on a continuous basis.

[0023] A "digital twin" of a sales area, created using positionable camera equipment and therefore representing only the moment of recording, i.e. being largely static, can be brought to life as a dynamic digital twin, or in other words, brought to life, by updating it with respect to events or changes in the relevant local part of the sales area.

[0024] Records of the portion of the sales area within the fixed-position camera device's field of view can be assigned to that portion of the sales area for which a different measure of updating of the digital image of the sales area is required. For example, for the initial portion-determined records of the fixed-position camera device, computer vision can be used to determine and store which areas of the digital image of the entire sales area the fixed-position camera device's records relate to. In this case, computer vision evaluates the fixed-position camera device's records to identify areas of the sales space within the fixed-position camera device's field of view based on the match of image content between the records of the variably positioned and fixed-position camera devices. This can be done based on the degree of match of detected details of the image content.

[0025] However, depending on the individuality of the recording area, this allocation can also be made on an ad hoc basis, and in particular if sufficient computing power and operational capacity is available on the part of the data processing device, the prior allocation for subsequent updates can be completely omitted and processing can be carried out entirely in ad hoc mode, which largely means that the recordings made by the fixed-position camera device are always allocated to the corresponding area as they occur, in a computer-assisted manner.

[0026] By repeating the software-based allocation at times relatively far apart from each other, it is possible to verify the prior allocation made at an earlier time, on the one hand allowing this allocation to be periodically checked and possibly automatically adjusted, and on the other hand providing the computing power and operational performance of the data processing equipment for the actual updating and visualization during these verification times while the operation is ongoing.

[0027] Optionally, it may be provided that this allocation is carried out using further technical means or measures. Thus, for example, as early as the recording of the variably positioned camera devices, the fixed-position camera devices can be photographed together and detected in the recording and possibly identified based on the detection and / or identification characteristics of the fixed-position camera devices. Since the optical and possibly also the digital photographing parameters of the fixed-position camera devices are essentially known and can therefore be stored in a data processing device, and since computer vision obtains an overview of the entire sales area at least while the photographing of the sales area with the variably positioned camera devices is in progress, the part of the sales area that can be photographed by the fixed-position camera devices can also be identified (i.e. calculated) or at least limited using computer vision based on the direction or orientation of the fixed-position camera devices detected using computer vision.

[0028] As operations progress, i.e., as soon as recordings from fixed-position camera equipment are obtained, the definition of parts of the sales area can be improved until the definition is finally ranked as reliable or sufficient.

[0029] Technically, the recordings of fixed-position camera devices can be mapped to the recordings of variable-position camera devices by applying various image processing techniques, such as feature points, neural networks, or classical computer vision, or even a combination of the techniques described here. If a successful allocation of a recording of a fixed-position camera device to an area of ​​a sales area, i.e., to at least one recording of a variable-position camera device, has already been achieved in the past, it is possible to rely on "historical data" representing this existing allocation in order to save computational power during future analysis. Advantageously, whenever a new recording of a fixed-position camera device is obtained, for example, immediately after a new individual image is recorded, this recording of the fixed-position camera device is allocated, i.e., located. Advantageously, whenever an update is carried out using the variable position camera equipment on the digital image of the sales area extending along its movement path, i.e. whenever an already existing master 3D location map is updated, a check is made to see whether the recordings of the fixed position camera equipment or the positions of the fixed position camera equipment are correct and still correct, or if not, whether they must be adjusted because the topology of the sales area has changed in the meantime, for example due to a change in the grouping of products on a rack, a change in the rack position or its orientation, or otherwise a fundamental reconfiguration of the product display areas (racks, baskets, etc.) in the sales area.

[0030] For this allocation, beacons installed in various locations within the sales area can also be used. The locations of the beacons can be known to the data processing device, i.e., can be stored there. At least some of the beacon signals should be receivable at a location within the sales area, so that the location of the beacon receiver can be estimated or at least limited, for example, based on the individual detection of the beacons from which the signals originate and the individual signal reception parameters for receiving the signals. If the fixed-position camera device and the variable-position camera device are equipped with such beacon receivers, the locations of the fixed-position camera device and the variable-position camera device can be aligned, so that the recordings can be allocated to each other according to their spatial correspondence.

[0031] During the course of operation, the variable-position camera equipment is moved along a planned, predefined, or otherwise accidental, path of movement, so it can be assumed that the generation of digital images of the sales area will be repeated for the entire sales area or only for a section of the sales area. This would make updates for that section of the sales area relatively long, or could be used to verify, or possibly adjust, the previously created digital images of the entire sales area or that section of the sales area. However, since the variable-position camera equipment is indeed essentially used to photograph a wide area of ​​the sales area, which is within the defined characteristics of the variable-position camera equipment, such a section of the sales area will typically be relatively small compared to the portion photographed using a fixed-position camera, which is typically the entire sales area, and much larger than the portion of the sales area.

[0032] In order to ensure timely real-time performance, it is advantageous for the updates based on the recordings of the fixed camera devices to be more frequent, in particular much more frequent, than the generation of digital images using the variable camera devices. In this case, it is essential that the photographing of a portion of the sales area using the fixed camera devices is performed independently of the photographing of the wider sales area by the variable camera devices. The term "independently" should be interpreted in this context as relating to both time and location. That is, the photographing of a portion of the sales area by the fixed camera devices can be performed independently in time from the photographing by the variable camera devices, which in turn is performed independently of the question of whether any real events are occurring in that portion of the sales area. The photographing of a portion of the sales area by the fixed camera devices is also performed independently in terms of the location and the position of the variable camera devices at any given time. The only exception is if the variable camera devices are simply moving through the photographing range of the fixed camera devices, for example, as a real event. This helps ensure that in the digital twin, the actual situation in that part of the sales area is or can be obtained in a timely manner relative to events in that part of the sales area, a process that is completely decoupled from captures by positionable camera equipment, which are usually not responsive in both time and location.

[0033] In this context, "much more frequent" can specifically mean that recordings using the fixed-position camera devices are carried out at regular intervals of a few minutes or even a few seconds. Also, "much more frequent" can mean that recordings of the fixed-position camera devices are always triggered or provided in response to an event within the camera device's field of view. This event can be, for example, the presence of a person, which is detected, for example, based on the image content or by means of an additional motion sensor in the fixed-position camera devices. However, this event can also be a change in the arrangement of products detected in the image content in the area of ​​the sales area captured by each fixed-position camera device. In this case, it has proven particularly advantageous for the fixed-position camera devices themselves to be equipped with the electronics and / or software required to identify or distinguish these events. This reduces the amount of data to be transmitted to the data processing device and the frequency of this data transmission, as well as the computing power required on the part of the data processing device. That is, only event-triggered recordings are sent to the data processing device. Unless the network bandwidth and the computing power of the data processing device are critical, the detection of events can of course also be carried out by the data processing device, in which case the different recorded data of the fixed-position camera device should be transmitted to the data processing device almost continuously in order to be able to react in a timely manner to the events to be detected.

[0034] In a first embodiment, the fixed-position camera device is a camera fixed to the product display equipment, in particular to the rack rails of the product display equipment, and from its position, according to its imaging range, the camera takes digital images of the part of the sales area that is within its imaging range.

[0035] The camera may have a fixed field of view, or may have a variable field of view because it is configured to zoom, or because its electronics are capable of further processing only one sector of the field of view. Such cameras may be fixed to different locations within the sales area, such as to a rack structure or deck in the sales area, or to an overhead support structure that may be deployable to support, for example, lighting fixtures or other electronic components, such as radio equipment.

[0036] However, advantageously in this connection, a camera, also referred to in technical terms as a rack rail camera, is used, which is configured to be mechanically, and possibly also electrically, fixed to the rack rail, from which position it can photograph the opposing rack structure in its rack aisle, together with possibly the conventional paper and / or plastic rack labels attached thereto, the electronic display devices arranged thereto and configured to display product and / or price information in an electronic manner, the products displayed thereon, and people and other moving or stationary objects in the rack aisle.

[0037] In a second embodiment, the fixed-position camera device is a camera whose shooting direction can be changed, and from its position, and possibly while changing its shooting direction, the camera takes digital images of the part of the sales area that is within its shooting range according to the shooting range given by the shooting direction at that time.

[0038] The camera may have a fixed field of view or may have a variable field of view because it is configured to be zoomable or because its electronics are capable of processing only one sector of the field of view further, and as already mentioned, the camera may be fixed to various locations within the sales area, for example on a rack structure, on the sales area deck or on an overhead support structure.

[0039] The variable imaging direction can be realized, for example, by an electric motor drive that regularly rotates the camera direction, and thus the imaging range direction, back and forth along a pre-set trajectory between two final positions. However, the imaging direction can also be remotely controlled, for example, by the data processing device, and indeed can be changed depending on the situation, for example, if this facilitates or provides a meaningful contribution to updates. In this way, the data processing device can, by software, target the imaging direction towards a part of the sales area if it detects that this is important for the real-time nature of the digital twin or because a number of activities that may be relevant to real-time nature are taking place there.

[0040] As a camera that can be changed in terms of its shooting direction, it is possible to advantageously use a surveillance camera, which is indeed often already installed in the sales area and which is integrated into the method, i.e. so that the data processing device has access to its surveillance video or still image data. This can be done in such a way that the camera sends the recordings it makes directly to the data processing device, or in such a way that the data processing device accesses the recording data that is stored using a surveillance recording server and from which it is retrieved.

[0041] As a camera that can be changed in terms of shooting direction, it is particularly advantageous to use a rotating / swivel head camera as disclosed in the description of the drawings, which, despite being fixed in position, is characterized by its shooting performance being as flexible as possible thanks to its rotating / swivel head.

[0042] It has proven particularly advantageous if this updating comprises detecting the movement of objects and tracking such moving objects in digital images of a portion of the sales area, in particular during recording by at least two fixed-position cameras, this being achieved by means of the aforementioned software which analyses the recordings of the moving objects.

[0043] Regardless of what type of camera is used as the fixed-position camera device to track the moving body in the records sent to the data processing device, the purpose of tracking the moving object may be, for example, to analyze the path of a shopping cart with respect to the frequency with which it moves along a predetermined route within a sales area, thereby enabling advantageous location estimations to be made for products and marketing measures, and also to detect areas within the sales area where undesirable occurrences occur, identify their causes, and derive suitable measures to prevent such occurrences in the future or to redirect them elsewhere.

[0044] In yet another aspect, it can be provided that this updating includes detecting people, so that people or other items can be tracked in the sales area or their movement patterns can be evaluated. The explanations already given above in connection with the detection of moving objects and the analysis of the paths or stasis occurrences that moving objects often take in the sales area also apply to this aspect. Another application example is, for example, tracking certain items, such as luxury or expensive items, from their location on the rack to the checkout area, either directly or in association with the person who removed the item from the rack, in order to investigate whether the item was actually paid for by that person. In this case, the tracking associated with the person can of course be carried out anonymously and can otherwise be developed in accordance with current data protection regulations.

[0045] As regards the temporal behavior of this update, it has already been mentioned that it is carried out at least more frequently, and in particular much more frequently, than the generation of the digital image or the slow updating of the digital image.

[0046] According to a first embodiment variant, this updating can be performed in a time-controlled manner, in particular periodically, for which purpose the fixed-position camera device can send in a time-controlled manner, for example regularly or periodically, a record of the part of the sales area which it has photographed to the data processing device.

[0047] Another embodiment variant is that this update is triggered automatically depending on the situation. In this regard, the fixed-position camera device can be equipped with an autonomous analysis function that allows it to send a recording of part of the sales area to a data processing device, which then decides whether to trigger an update. This analysis function can be based, for example, on a motion sensor installed in the camera detecting movement in front of the camera, a temperature sensor installed in the camera detecting the temperature or temperature changes around the camera, a wireless module installed in the camera receiving a special wireless signal or determining from this signal the approach or departure or approach angle of a wireless transmitter, etc. It can also be provided that the fixed-position camera device sends its recording in a time-controlled manner to a data processing device, which then decides whether a situation for updating has occurred.

[0048] Furthermore, it has proven particularly advantageous for this updating to include detecting electronic display devices. This detection can be performed directly by a fixed-position camera device or the detection results can be sent to a data processing device for further processing. This detection by the data processing device can also be performed based on recordings sent from the fixed-position camera device. Detecting electronic display devices allows the electronic display devices to be precisely located in the digital twin, and, if necessary, the contents of their screens can also be optically photographed and evaluated, and, if necessary, optical signaling by LEDs can also be photographed. In this way, the actual location of the electronic display devices, as well as their status, which, if necessary, is displayed by the contents of their screens or by signaling using light-emitting diodes, can be depicted directly in the digital twin.

[0049] The positioning of electronic display devices, often realized as Electronic Shelf Labels (ESLs), technically known as "Electronic Shelf Labels" or ESLs for short, but which can also be realized as video rack rails or video panels, is based on the fact that a fixed-position camera device captures not only the electronic display device but also its surroundings, which are then inserted into a digital image of the sales space generated using a variable-position camera device. This can be done in such a way that the recording of the fixed-position camera device overlaps the recording of the variable-position camera device, by matching at least parts or areas with its contents.

[0050] Similarly, it can be provided that a fixed-position camera device is photographed together with a variable-position camera device, so that its position within the sales area is known, and subsequently, since its photographing range is basically known, positioning of the electronic display device photographed using the fixed-position camera device can be performed.

[0051] From another perspective, it is advantageous for this update to detect the presence or absence of products. In this regard, recordings from fixed-position camera devices are searched for patterns that match products, and then those products are placed in the digital twin. In this regard, detection from electronic display devices can also be usefully incorporated, since they reliably represent the products. To a certain extent, depending on the optical conditions occurring during imaging with variable-position camera devices, quantitative statements can be made about the presence of products in the sales area, and in any case, about the absence of products at a given location in the sales area. Therefore, if the coverage of fixed-position camera devices is sufficient, a dynamic three-dimensional realogram can be created that represents a complete picture of the actual presence of products in the sales area, both in terms of availability and location. By automatically comparing the digital planogram, which shows the desired state of product availability and product placement, with the digital realogram, which also shows the actual state of product availability and product placement, necessary measures can be derived and reported to employees contributing to the realization of the planogram.

[0052] It has proven particularly advantageous to carry out a real-time update when at least one of the previously listed events occurs, i.e. the occurrence of each event does not only result in the recording of the fixed-position camera equipment being saved for later use, but rather the update is carried out immediately, within the limits of the available computing power and availability of the data processing equipment, in order to guarantee an instantaneous real-time digital image of the sales area, which opens up new possibilities for the operator to manage the sales area.

[0053] In this way, the operator of a sales area can use the visualization equipment to track in real time how the situation in the part of the sales area he is observing changes over time and what events are occurring there, such as product shortages, product replenishment, customer flow, customer congestion, etc. This is not limited to a single section of the sales area (which may be freely definable by changing the visualization parameters associated with the software), but can also be applied to an overview of the entire sales area. Furthermore, any event that triggers an update or occurs only within the scope of the update can be visually emphasized, for example, by color highlighting of the background of the image, color framing, or a dynamic signal display at that position of the image.

[0054] In the following, aspects related to positionable camera devices are addressed.

[0055] According to the disclosure at the beginning of the present application, the positionable camera device can be configured as a self-propelled type, for example as a self-propelled robot. Advantageously, the photographing of the surroundings of the travel path is carried out using a passive positionable camera device. This makes the photographing of the basic structure of the sales area significantly more advantageous, easier to realize and more flexibly configured, in particular without the problems and risks listed at the beginning of the present application.

[0056] A passive camera device should not be construed as a camera attached to a self-propelled robot that is moved through a sales area according to a path of travel defined by the robot's logic or remote control. Rather, this passive movement is movement that results from the action of a person, who either directly moves the positionable camera through the sales area, or indirectly because the positionable camera device is affixed to an object, such as a shopping cart, that is indirectly moved by the person.

[0057] In a first embodiment, the passively positionable camera device can be a 360° camera that captures images of the surroundings of a moving path in all directions. The 360° camera can be attached to a shopping cart, for example, to an upwardly protruding pole, advantageously at its end, so that the person passively moving the camera does not significantly obstruct or dominate the 360° camera's recording. However, the 360° camera is preferably attached to a person's body or clothing. Preferably, the camera is mounted at the head or shoulder position to obtain an all-around view that is as unobstructed as possible. Such a passively moving configuration allows the camera to be used at all times to capture images of the surroundings of a moving path as the person moves through the sales area. Indeed, since the 360° camera is, by definition, configured to capture images of the surroundings in all directions, it does not matter in what direction the person is moving within the sales area or in which direction the person is standing or moving. A 360° camera can either continuously capture its surroundings, excluding the area where people are hiding, or it can capture its surroundings only when its integrated motion sensor detects movement, i.e., it can therefore capture its surroundings starting from the point of view of the 360° camera moving along a path of movement.

[0058] In a first embodiment, the surroundings of the travel path are captured using a positionable camera device mounted on a shopping cart. This embodiment also uses a passive camera device, since the shopping cart is moved by a person moving within or through the sales area or by a customer moving through the sales area. In this regard, as mentioned above, a 360° camera mounted on the end of a wand can be used. However, experience has shown that wands are often used improperly, for example, to hang pieces of clothing, which completely obscure the 360° camera and thus cause it to malfunction. Optionally, the length of the wand can be selected to prevent this from happening, but this may lead to aesthetic criticism of the design. Although the wand is indeed provided solely to carry very light camera equipment and is essentially constructed to be relatively fragile and easily broken to serve this purpose, such a solution is also prone to damage, since people often tend to grab the wand and move the shopping cart. In any case, the safe use of the wand involves considerable additional costs.

[0059] It has therefore proven particularly advantageous to film the surroundings of the travel path from the basket of a shopping cart, where the positionable camera is particularly protected, and since the bottom of the rack is usually mounted with a slight downward incline towards the front, from this relatively low position the different levels of the rack can be seen without any problems.

[0060] Furthermore, in this connection, it has proven particularly advantageous to photograph the surroundings of the travel path through openings in the basket or its walls of the shopping cart, in particular through openings between the bars of the basket. In this case, the camera equipment is installed in a safe position. The walls of the basket or the bars of the basket, which are usually made of metal and designed for maximum load, provide the necessary protection against improper handling by the person moving the shopping cart and collisions with hard objects, so the equipment cannot be damaged there. In other words, the existing mechanical structure of the basket is utilized in two ways: on the one hand, for protection, and on the other hand, for supporting the camera equipment.

[0061] To prevent the camera device's coverage from being obscured by the person pushing the shopping cart, the camera device is not fixed to the basket portion facing the person pushing the shopping cart. The camera device can be fixed, for example, to a side portion of the basket to capture the surroundings of the path of movement from there. However, if the camera device is fixed to one of the side portions, it can only capture the surroundings of that side, and the coverage from there may sometimes be significantly limited by other objects located relatively close to the rack or along the rack aisle. Therefore, it is advantageous to capture the surroundings of the path of movement from a position at the front of the shopping cart. There, the camera device usually has the widest coverage and can therefore be used for wide-area coverage. Also, there, the camera's coverage is not obstructed by objects in the basket.

[0062] Particularly advantageously, the positionable camera device comprises at least one recording stage equipped with a stereo camera for recording the surroundings of the travel path. The recording stage should be understood as including at least the optical components and electronics required to generate raw video and / or still image data representative of the recorded scene, which is generally known to those skilled in the art. The stereo camera allows recording the surroundings in a stereoscopic image, which in turn facilitates evaluation of the three-dimensional structure by image processing software, e.g., computer vision.

[0063] Advantageously, the variably positionable camera device comprises at least a first and a second imaging stage, and advantageously also a third imaging stage, each imaging a separate spatial sector around the path of movement, which means that each of these imaging stages has its own optical imaging system oriented according to the spatial sector to be individually imaged.

[0064] It has proven particularly advantageous for the positionable camera device to have three imaging stages at the front of the shopping cart, with the first imaging stage imaging a first spatial sector diagonally forward to the left, the second imaging stage imaging a second spatial sector diagonally forward to the right, and the third imaging stage imaging a third spatial sector in the center front. This allows the recording of each imaging stage to individually optimize the imaging, and therefore the information provided. In this way, the side-facing imaging stages can be optimized to focus on the contents of the racks, and since they usually pass relatively close to the racks, they do not require a very deep imaging range. In contrast, a imaging stage facing toward the center front can be configured with a fairly wide imaging range so as to be able to "see through" the rack aisles as far as possible.

[0065] In this connection, it has proven particularly advantageous to arrange the first imaging stage, which photographs the left diagonal front, at the front right of the shopping cart, and the second imaging stage, which photographs the right diagonal front, at the front left of the shopping cart. Such an intersection of the imaging directions at the front of the shopping cart maximizes the distance between the optical components of each imaging stage and the structure or object to be photographed, thereby achieving the widest possible imaging range for each imaging stage.

[0066] In order to obtain a continuous image of the surroundings of the positionable camera devices along the front of the shopping cart, it has proven particularly advantageous to configure two of these image capture stages so that the spatial sectors captured by them overlap at least slightly. Thus, the image capture area facing the left-hand diagonal front direction can overlap slightly with the image capture area facing the center-front direction. Similarly, the image capture area facing the right-hand diagonal front direction can overlap slightly with the image capture area facing the center-front direction. These slight overlaps allow computer vision to identify connections or overlapping areas in each capture and process the captures in a stitched manner.

[0067] The variable-position camera device can be realized as a component of the shopping cart, i.e., it can be pre-integrated into the shopping cart at the factory. However, particularly preferably, the variable-position camera device is a retrofit part that can be fixed using a fixing mechanism that is appropriately adapted to the particular type of shopping cart. This has the advantage that the invention can be installed on millions of conventional shopping carts in use, in order to widely apply the invention with as little investment as possible. Particularly preferably, the variable-position camera device is integrated into a flat, plate-like housing, which is fixed to the front of the shopping cart from inside the basket of the shopping cart using the aforementioned fixing mechanism. This fixing mechanism holds the variable-position camera device to the structure of the basket, for example, by a fixing plate that is screwed to the plate-shaped housing from the outside of the basket, so that part of the basket structure is sandwiched between the fixing plate and the housing. The fixing mechanism can also be a plug-in mechanism that allows for easy insertion into the inner wall of the basket. The optical components of each imaging stage can be arranged to fit the shape of the shopping cart, or arranged in a variable shape, so that they can "see through" the opening of the basket without being obstructed by the basket structure, i.e., so that the optical path of each optical component is affected as little as possible by the basket.

[0068] In yet another embodiment, the retrofit part can be configured so that the optical components can see beyond the upper edge of the shopping cart. This allows for a more versatile retrofit part, where the position of the optical components is decoupled from the existing structure of the basket. The fixation of such a retrofit part can also be performed independently of the position of the optical components below the upper edge of the basket. Significantly, the same applies to a retrofit part that is configured to be fixed to the underside of the front edge of the basket with a corresponding fit, from which the optical components of the imaging stage can be positioned without being hindered by the structure of the basket.

[0069] In summary, it should be noted that both the system and the method described above are characterized by the following set of features:

[0070] The measures described above allow for the fully automated generation of a digital map of the sales area, also known in English as a "vector floor map," which can depict the sales area in two or even three dimensions.

[0071] The fixed-position camera devices are photographed together when generating the digital image, so their positions are known in the digital image and therefore also in the actual sales area.

[0072] The rack aisles can be detected automatically, and in particular the data from different camera devices along the rack aisles can be merged together, essentially making it possible to capture the entire situation in the rack aisle at any given time.

[0073] Portions of the sales area that cannot be photographed by the fixed-position camera devices can be detected automatically, either by detecting the absence of a suitably positioned fixed-position camera device as early as the time of digital image generation, or by determining that an update is apparently not possible for those portions of the sales area, since a loss of suitable image data has occurred, i.e., the records of the fixed-position camera devices associated with those portions are missing.

[0074] Image data obtained using variable and fixed position camera equipment no longer only represents a snapshot for analysis (e.g., inventory status on a rack), but also contains true location information, as digital images generated over a wide area of ​​the sales area are updated to correspond to that portion of the sales area.

[0075] This again allows you to view the entire rack using a suitable playback device (screen, VR headset, etc.) and see what is happening in real time.

[0076] Furthermore, the measures described above allow for precise placement of merchandise and promotions within the sales area.

[0077] Also, three-dimensional realograms and planograms can be visualized, or contrasts or comparisons of realgrams and planograms can be visualized, possibly highlighting relevant differences.

[0078] The digital imagery can also display location-specific promotional activities.

[0079] For example, enhanced data visualization at the store level, such as so-called "heat maps," can be integrated to visually highlight features in sales areas in a current digital image, thereby pointing out "out of stock" areas that urgently need restocking, thereby pointing out the best selling racks, and so on, or to display other "key performance indicators," with or without reference to location.

[0080] The operation and maintenance of this system can also be simplified. In this way, digital images can be used to display devices with low battery levels, especially ESL or camera devices. Similarly, the signal strength or signal quality of the wireless signals used for data transmission occurring at each camera device's location can be displayed, i.e., for example, a WLAN signal map can be integrated into the visualization of the sales area.

[0081] As mentioned above, existing surveillance infrastructure (surveillance cameras, networks, video data memories or still image data memories, etc.) can be integrated into the above-mentioned measures, i.e., the existing infrastructure can also be utilized for the purposes of the above-mentioned methods.

[0082] Additionally, queue visualization, customer tracking using route maps or the aforementioned "heat maps" can be mixed into the sales area visualization, or these data can be stored for subsequent evaluation or analysis.

[0083] It should be further noted here, quite generally, that the data processing device can be programmed to drive the ESL to display product and / or price information. Status information can also be received from the ESL and processed. Similarly, LEDs available in the ESL can be driven, if necessary. The communication network also used for communication with the camera device can be used for communication with the ESL. However, a separate and distinct communication network can also be used for communication with the ESL. The necessary measures (devices and methods, hardware and software) associated with the operation and maintenance of an ESL are well known to those skilled in the art.

[0084] Finally, it should be noted that, generally speaking, the aforementioned electronic equipment and devices naturally comprise electronic components. These electronic components can be composed of discrete or integrated electronic components, or a combination of both. Microcomputers, microcontrollers, and application-specific integrated circuits (ASICs) can also be used, in some cases, in combination with analog or digital electronic peripherals. Many of the aforementioned functions of these devices are implemented using software running on the processors of the electronic components (in some cases, working in cooperation with hardware components). Devices configured for wireless communication usually have an antenna arrangement for transmitting and receiving radio signals, as well as a modulator and / or demodulator, as part of a transceiver module. Furthermore, electronic devices can have an internal power source, which can be realized, for example, by a replaceable or rechargeable battery. These devices can also be powered in a wired manner via an external power grid or by "Power over LAN." Similarly, wireless-based power supply via "Power over WiFi" can be provided. Battery-powered electronic display units advantageously have energy-saving electrophoretic screens.

[0085] These and further aspects of the present invention will become apparent from the drawings described below.

[0086] The present invention will be explained in more detail below based on examples with reference to the accompanying drawings, but the present invention is not limited thereto. [Brief explanation of the drawings]

[0087] [Figure 1] Partial view of a shopping cart basket with a repositionable camera device mounted in the front of the basket [Figure 2] Block diagram of the electronic components of a position-adjustable camera device with three imaging stages [Figure 3] Schematic diagram of the shooting range of the three shooting stages of the position-adjustable camera device [Figure 4] Schematic diagram of three shooting ranges for rack aisles [Figure 5A] Schematic of realistic still image recording of racks along a rack aisle using three imaging stages [Figure 5B] Schematic diagram of realistic still image recording of products arranged on racks along a rack aisle using three imaging stages. [Figure 5C] Schematic of realistic still image recording of electronic display devices along a rack aisle using three imaging stages. [Figure 6] Schematic of a sales area with three shopping carts equipped with position-adjustable camera devices. [Figure 7] Schematic diagram of a rack rail camera used as a fixed-position camera device [Figure 8] Flowchart diagram of a method for creating a digital image of a sales area [Figure 9] Schematic diagram of yet another implementation of a fixed-position camera device. DETAILED DESCRIPTION OF THE INVENTION

[0088] Figure 1 illustrates a basket 1 of a frequently used prior art shopping cart 2, at the front of which is a positionable camera device 3 mounted inside the basket 1. The basket 1 is made up of a metal rod, hereinafter referred to as rod 4 for short.

[0089] The positionable camera device 3 is an add-on for the shopping cart 2. It is fixed to the basket 1 by means of two fixing plates 5 arranged on the outside of the basket 1, which are screwed to the plate-shaped housing 6 of the positionable camera device 3 by means of a second screw 7 (only the left fixing plate 4 is numbered). The positionable camera device 3 comprises three stereo image capture stages 8, 9, 10, hereinafter referred to as capture stages 8, 9, 10 for short. The stereo optical components 11, 12, 13 (see also the reference numerals 11, 12, 13 in FIG. 2 ) arranged for these image captures, or the individual image capture fields realized by each optical component, are essentially not obstructed by the bars 4 of the basket 1. That is, each stereo optical component figuratively "sees through" between the bars. It should be noted that in this case, the stereo optics of the central imaging stage 10 are aligned horizontally, while the stereo optics of the two outer imaging stages 8 and 9 are aligned vertically, which facilitates stereo resolution in each orientation.

[0090] To enable photography unhindered by the rods 4 of the basket 1, the first stereo optical element 11 of the first photography stage 8 is mounted on a first positioning arm 8A that can be flexibly pulled out from or inserted into the central housing 6, so that it can be variably positioned at the front of the shopping cart 2, taking into account the rods 4 or any gaps that may occur between them. The same applies to the second stereo optical element 12 of the second photography stage 9, which is mounted on a second positioning arm 9A that can be flexibly pulled out from or inserted into the central housing 6.

[0091] The first image capturing field (reference numeral 20 in FIG. 3) of the first imaging stage 8 is directed diagonally forward and left, the second image capturing field (reference numeral 21 in FIG. 3) of the second imaging stage 9 is directed diagonally forward and right, and the third image capturing field (reference numeral 22 in FIG. 3) of the third imaging stage 10 is directed toward the center and front. Three image capturing fields, hereinafter referred to as capturing fields 20, 21, and 22 for short, which actually have a spatial extent, i.e., are approximately equal to a spatial sector, are visualized as shown in FIG. 3 and described below.

[0092] Figure 2 illustrates the functional groups of the positionable camera device 3 in a block diagram. The three imaging stages 8, 9, and 10 are indicated with their respective stereo optical components 11, 12, and 13, as well as their orientations as described in the previous paragraph. These imaging stages 8, 9, and 10 are digital stereo cameras that output image data BD from their recordings to a processing device 15 implemented by a microcontroller and software running on it via a bus 14 configured for transmitting digital data and / or control signals. This processing device 15 is used to (pre)process and / or evaluate the image data BD and, as the case may be, to control communication with external devices, such as access points or network components of an IoT network (here, IoT stands for "Internet of Things"), which transmit the preprocessed image data BD or the evaluation results obtained from the evaluation. For physical communication with external devices, the camera device 3 has a first wireless communication stage 16 equipped with the necessary active and passive electronic components (commonly called a transceiver), which is indicated only by an antenna arrangement 17.

[0093] Furthermore, for its power supply, the positionable camera device 3 comprises a power supply device 18, which is realized by means of a rechargeable battery 19 and is configured to provide a power supply voltage VCC relative to a reference potential GND. To charge the battery 19, the power supply device 18 can use various technologies, such as "Power over WiFi", which transmits energy using the WiFi signal, or a photovoltaic panel integrated in the housing 6 for "energy harvesting" from ambient light.

[0094] FIG. 3 shows a shopping cart 2 equipped with a position-variable camera device 3. Furthermore, in this case, starting from the front of the shopping cart 2 or three camera stages 8, 9, and 10 arranged there, a first camera range 20 of the first camera stage 8, a second camera range 21 of the second camera stage 9, and a third camera range 22 of the third camera stage 10 are displayed. In this case, the edges of at least the camera ranges 20 and 22 and the overlap of the edges of the camera ranges 20 and 21 are clearly visible near the shopping cart 2, making it easy to combine images created using each camera stage 8-9 into a kind of "panoramic image" of the scene in front of the shopping cart 2. For the positions of these three camera stages 8, 9, and 10, please refer to FIG. 1.

[0095] 4 shows a shopping cart 2 positioned between two racks 23, which is typical of a sales area 27 of a retailer operating, for example, a supermarket, where the racks 23 are lined up on either side of a rack aisle. The shopping cart 2, and therefore the position-adjustable camera equipment mounted on the shopping cart 2, is typically moved passively by a person (male / female customer). The racks 23, or more precisely, the rack bottoms or rack faces 24 of the racks 23, are sparsely stocked with products 25, but for clarity, not all of these are labeled. In this case, it can be clearly seen that the first imaging range 20 oriented diagonally to the left and the second imaging range 21 oriented diagonally to the right allow for an almost frontal view of the racks 23 arranged therein right up to the rack bottom 24 and therefore for very accurate detailed images of objects and structures therein, while the third imaging range 22 oriented in a central forward direction allows for a bird's-eye view of both the arrangement of the racks 23 along the rack aisle and the progression of the rack aisle in front of the shopping cart 2.

[0096] 5A-5C illustrate depictions, i.e., image captures, of the structure along the rack aisle using the three aforementioned imaging stages 8, 9, and 10. Thus, FIG. 5A illustrates the arrangement of racks 23 on the left side of the rack aisle, i.e., starting immediately in front of the shopping cart 2, as captured using the first imaging stage 8. Furthermore, FIG. 5C illustrates the arrangement of racks 23 on the right side of the rack aisle, i.e., starting immediately in front of the shopping cart 2, as captured using the second imaging stage 9. FIG. 5B illustrates the progression of the lateral racks 23 along the rack aisle in front of the shopping cart 2 until they reach a laterally extending rack 23, which then curves to the right in front of the laterally extending rack 23. It can be clearly seen that each pair of adjacent records of these racks 23, i.e., the pair of records according to FIGS. 5A and 5B and the pair of records according to FIGS. 5B and 5C, both illustrate a common area that allows the individual records of FIGS. 5A, 5B, and 5C to be linked together. That is, using the three camera stages 8, 9, and 10, the entire situation in front of the shopping cart 2 is captured virtually three-dimensionally, specifically between the right and left camera limits as well as between the top and bottom camera limits within the available camera ranges 20-22. These captured digital images show, in addition to the rack 23, the rack bottom 24, which is labeled only on its outer edge in Figures 5A and 5C, the products 25 displayed on the rack bottom 24, which is labeled only once in each of Figures 5A-5C, and an electronic display device 26, technically known as an "Electronic Shelf Label" or ESL for short, which is attached to the front edge of the rack bottom, typically on a rack rail incorporated therein, and which, for ease of viewing, is labeled only in a few places in the lower area of ​​each of Figures 5A-5C (it is known per se and therefore not explicitly labeled). However, these ESLs 26 are located at corresponding positions on the rack rails for each item 25 for all items 25, which is not fully visualized in order to avoid over-embellishing Figures 5A, 5B, and 5C.

[0097] The image data BD generated using the imaging stages 8, 9, 10 is output from the variably positioned camera device 3 using its wireless communication stage 16 for further processing, as will be discussed in more detail below. However, it should be noted here that further image content is continually obtained by the passive movement of the variably positioned camera device 3 and output using the wireless communication stage 16 in a substantially continuous sequence.

[0098] What has been said with respect to this one shopping cart 2 naturally also applies to other shopping carts 2, which are equipped with positionable camera devices 3 and moved in a transitive manner through the sales area 26, as shown in Figure 6. In this configuration, images of the zone in the sales area 26 in front of each shopping cart 2 are captured almost simultaneously (depending on network delays) by the respective positionable camera devices 3, and the resulting image data BD is output. The image data BD generated successively while the different positionable camera devices 3 are moved in a transitive manner are sent via a wireless network from an access point 28 (for reasons of simplicity, only a single access point 28 is shown, but in reality several access points are distributed throughout the sales area) to a data processing device 29, which is wired by means of a LAN network (here, LAN stands for "local area network") and consists of a high-performance server with application-specific image processing software, where it is assembled into a three-dimensional overview of the sales area 27. As soon as the last region of the sales area 27 has also been photographed using the positionable camera device 3, a complete digital image of the sales area consisting of all structural elements or objects (racks, rack bases, products, ELS, etc.) can be created, which in the Applicant's terminology is called a "master 3D location map", since this digital data structure stores a complete picture of the objects and their three-dimensional arrangement within the sales area 27. This master 3D location map is essentially a static digital representation of the sales area, only insofar as this representation has a relatively modest dynamic nature, in that further movements by one of the transitive forms of shopping cart 2 are used to update the master 3D location map along each movement path.

[0099] To provide real-time capabilities to the essentially static digital twin of the sales area, a so-called rack rail camera 35 is used as a fixed-position camera device. Figure 7 shows such a rack rail camera 35 in front view. On its front side, a lens system 36 of a photoelectric image capture system (not shown in detail, collectively referred to as a "photography stage") is visible, which electronically captures images of the scene within the capture range. Also visible in front of the system is an LED 37 for sending an optically perceptible signal. Also visible in front of the system is a motion sensor 38 arranged to detect movement in front of the rack rail camera 35, i.e., in front of the rack 23 to which the rack rail camera 35 is attached. The image capture system, LED 37, and motion sensor 38 are connected to a microcontroller (not shown in detail) of the rack rail camera 35, which controls the functions of the rack rail camera 35.

[0100] To avoid embellishment, the rack rail cameras 35 fixed to the racks 23 are not explicitly shown in the other drawings. However, they are fixed to the rack rails of the racks 23 in the same manner as the ESLs 26, and unlike multiple ESLs 26 per rack 23, typically only a single rack rail camera 35 is provided, and this camera is mounted in a position on the rack 23 where it can photograph, i.e., digitally depict, almost the entire rack 23 on the opposite side of the rack aisle. These have a configuration similar to that of the variable position camera device 3 shown in Figure 2, but are provided with only a single photographing stage, and as mentioned above, only the lens system 36 therein is visible as an external feature.

[0101] This energy supply can be done in the same way as the ESL 26, i.e., for example, by rechargeable or replaceable batteries, or else by supplying power and / or data in a wired fashion via the rack rails themselves.

[0102] Similar to the configuration of the positionable camera device 3, the rack rail camera 35 has electronics with a second wireless communication stage (not shown) by means of which image data generated by the rack rail camera 35 is transmitted to the data processing device 29. The aforementioned microcontroller of the rack rail camera 35 controls the internal processing sequence and communication with external devices via the second wireless communication stage.

[0103] In the data processing device 29, based on the image content of the records made by each rack rail camera 35, these records are assigned to the relevant portion of the digital image of the sales area and update this portion, for example by replacing the data previously characterizing this portion. The updating of this portion of the digital image can be carried out taking into account the aspects enumerated in the general description or independently of them.

[0104] The rack rail camera 35 is adapted to the respective rack rail used, i.e., configured in approximately the same shape as the fastening means of the ESL 26, for attachment to the rear side (not shown) of the rack rail in question. As such, a pin 39, visible in Figure 7, can also be provided on the upper right edge, by means of which it is possible to establish a connection with the rack rail that cannot be released without additional tools.

[0105] FIG. 8 illustrates an implementation of a method 30 for creating a real-time digital image of a sales area 27 according to the present invention.

[0106] The method 30 begins with a first method step 31, in which the data processing device 29 starts the processing of image processing software and moves at least one variably positioned camera device 3 through the sales area 27. In a second method step 32, the recordings of the variably positioned camera device 3 made along the path of movement of the variably positioned camera device 3 are used for the computer-processed generation of a digital image of the sales area 27, or of only a section or partial area of ​​the sales area 27 that can be photographed along the path of movement, and wait to be updated until most or all of the sales area 27 has been completely digitally depicted once.

[0107] Thereafter, in a third step 33 of the method, image data from the fixed-position camera devices 35 is received by data processing devices 29 to update respective portions of the digital image of the sales area 27 .

[0108] Optionally, this update can occur as early as to those parts of the sales area 27 for which digital images have already been created, before the entire sales area 27 has been digitally rendered.

[0109] Optionally, at any time, a branch can be made from the third step 33 of the method to the second step 32 of the method, for example if an update of one section of the sales area is carried out based on the recording of one of the positionable camera devices 3. However, this update can also be processed simultaneously with an ongoing update.

[0110] The method 30 ends in a fourth method step 34, for example, by the image processing software terminating processing.

[0111] Furthermore, we would like to highlight a specific embodiment of a fixed-position camera device that can be used in connection with the method 30 described here. This is a rotating and swivel-head camera 40, shown in FIG. 9 and referred to below as camera 40 for short. This camera includes, inter alia, a combined rotating and swivel head 41 carrying a camera system for digital image capture. This rotating and swivel head 41 is divided into a rotating part 42 and a swivel part 43. This rotating part 42 can rotate 360° (in both directions) relative to a base 44, parallel to the plane of the base 44, according to a first arrow 45. At its end opposite the base 44, the rotating part 42 carries in a fork-shaped recess a swivel part 43, which can swivel 180° (in both directions) relative to the base 44, according to a second arrow 46. This rotating and swivel movement can be controlled by the electronics of the camera 40 in an electrically driven manner.

[0112] This swivel 43 is equipped with a triple lens system 47, hereinafter referred to as lens system 47 for short, which comprises three 12 megapixel optical elements with different focal lengths of 120 mm, 78 mm and 45 mm, providing a 4x zoom, which can be used selectively or collectively in conjunction with their different imaging parameters to photoelectrically capture parts of the sales area.

[0113] Furthermore, the pivoting unit 42 is equipped next to the triple lens system with a DSGVO compatible (GDPR compliant) motion sensor 48. Using this motion sensor 47, the camera 40 can autonomously (by software control) distinguish between stationary and moving objects within its field of view, which facilitates discrimination between objects and people within its field of view.

[0114] The pivoting part 43 is provided with RGB-LEDs (Red Green Blue Light Emitting Diodes) 49 for transmitting optically perceptible signals.

[0115] The triple lens system 47, the motion sensor 48 and the RGB-LED 49 are oriented towards each other in a plane along a gently sloping area of ​​the normally approximately cylindrical pivoting part 43 so that they are oriented together towards the object or space sector based on a pivoting and rotating movement.

[0116] The electronics of the camera 40 (not shown in detail) are designed to be extremely energy-efficient and include, among other active and passive electronic components, a microcontroller that controls or provides different functions. To power the camera 40, an internal battery, in particular a rechargeable or replaceable battery, can be provided. The USB-C port 50 serves as a charging port for charging a rechargeable battery or connecting an external battery. The POE port (Power over Ethernet) 51 serves both to connect to a wired local network and to provide power via this network. Optionally, the camera 40 can be equipped with an LTE mobile radio module (not shown) for mobile wireless communication. An IoT module (not shown) can also be installed to enable sensor and related connectivity features.

[0117] This camera 40 is advantageously suited for selectively updating the part of the sales area 27 it photographs in a broad and / or focused manner. The camera can be directed, possibly autonomously, towards any spatial region around it, which is freely selectable thanks to its rotation and swivel capabilities and flexible optics, making it ideally suited for flexible monitoring of its surroundings. This means that the camera can be directed towards different parts of the sales area 27 and photograph events there, where real-time requirements arise in the digital image (as this is detected autonomously, for example, by means of a motion sensor 48), in a much more flexible manner than the fixedly installed rack rail camera 35.

[0118] The mounting of the camera in the sales area 27 is advantageously, but not necessarily, on the ceiling or other overhead structure from which the camera 40 can take a wide view of the sales area 27 or parts thereof. To allow as much flexibility as possible in its mounting on different objects, the camera is provided with an installation or fixing adapter system 52 on the side of the base 44 opposite the rotating and swivel head 41.

[0119] The image data provided by the camera 40, possibly together with the results of any pre-processing or analysis carried out by the camera 40, is sent via the access point 28 to the data processing equipment 29 as described above, where it is used to update the digital image of the sales area 27 with respect to the part of the sales area 27 photographed by the camera 40.

[0120] However, camera 40 can also be used to achieve additional applications, which will now be discussed.

[0121] The camera 40 can be programmed, for example, based on software, to use its rotating and swiveling head to take digital images of the part of the sales area that is within the range of its rotating and swiveling movement, searching for or analysing events that require increased attention or that should be transmitted to the data processing device 29 for external evaluation or analysis.

[0122] These events can include, for example, the light signal of the LED of the ESL 26, i.e., a constant or modulated light output from the LED. For example, if the ESL 26 transmits its identification code using a modulated light signal, software in the camera 40 can evaluate this identification code and provide it for further data processing in the camera 40 or the data processing device 29. In this case, the light signal is captured using a lens system 47, which is directed toward the ESL 26 using the rotating and pivoting head 41 when the light signal occurs within the camera 40's field of view. The camera 40 can further be programmed to create at least two images (still images or video) with two different focal lengths using the three-stage lens system 47. This allows the digital image to capture a sufficient area around the ESL 26, allowing the ESL 26 to be positioned as accurately as possible within the digital twin. At the same time, the other digital images are zoomed in correspondingly deep into that part of the sales area, so that the greatest possible variety of details can be guaranteed for visualizing that part of the sales area using the digital twin.

[0123] The light signals can have various meanings, allowing the ESL 26 to communicate its identity and / or status, such as a low battery or a service request due to other technical or logistical conditions, and can also make the ESL 26 more visible, making it easier for customers and concession staff to find.

[0124] However, the camera 40 can also be equipped with a radio system capable of receiving and evaluating so-called beacon radio signals. This radio system of the camera 40 can, for example, be configured to detect the approaching (particularly the angle of approach (technically referred to as the "angle of arrival")) and the departing (particularly the angle of departure (technically referred to as the "angle of departure")) of a radio beacon (beacon transmitter). Such a radio beacon can, for example, be realized as a component of a personal digital assistant, or PDA for short. A person moving through the sales area can carry this PDA with them for various purposes and use it for the tasks they have to perform. However, the radio beacon can also be integrated into a special shopping cart, forklift, or similar device that a person uses or carries with them.

[0125] Equipped with a PDA, a customer's shopping list can be sent to a staff member, who in this case acts as a store picker, who then moves through the sales area and collects the items 25 included in the list. As soon as the beacon radio signal is received by the camera 40 and identified as such by the store picker, and optionally also transmits its identifier, the camera 40 autonomously points itself toward the store picker (possibly by also changing the orientation of the lens system 47), tracks the picker within its viewing range, detects that the picker is in front of a given rack 23, and then photographs the picker removing from the rack 23 the items 25 listed on the shopping list. By photographing the relevant part of the sales area, the digital image can be used by the data processing device 29 to identify the items 25 removed from the rack 23 and update the shopping list displayed on the PDA accordingly. This allows for the photographing of that portion of the sales area (as well as the resulting updates to the digital twin) to be virtually continuous, so that, for example, an item 25 can be photographed as it is removed, observed, and then returned to the rack 23. Such a process may occur when the store picker is not sure whether he or she has found the correct item. Detection of such a process, and the resulting detection that the store picker has in fact picked up the wrong item, can be used to activate the data processing device 29 to flash the ESL 26 corresponding to the correct item 25, thereby facilitating the store picker's selection of the correct item 25 from among similar items 25 in the surrounding area. This allows for real-time updates of both the status of the items in the sales area and the status of the shopping list on the PDA.

[0126] The situation is similar when replenishing the racks 23 with products 25. The staff member carrying the PDA is photographed by the camera 40 as intended, and the replenishing process is precisely photographed together with the camera 40. As soon as the staff member has completed the replenishing process, this is automatically recognized solely based on the image data of the camera 40, and the current status can then be instantly revealed by the image in the digital twin.

[0127] When beacons are used in a sales area, for example for direction or navigation purposes, the camera 40 can also serve to activate only those beacons actually required for the purpose within the entire sales area, in the vicinity of the staff member who is photographed by the camera 40 as described above and who uses a PDA to point in the direction of the beacons placed in the sales area.

[0128] By using the camera 40, which can be flexibly oriented and flexibly zoomed, it is possible to dynamically track (particularly by pointing the camera 40 in different directions) whether a male / female customer is staying in front of any rack 23 within its variable shooting range and what they are actually doing there (from just looking at the products 25, taking the products 25 and putting them back again, taking the products 25 and putting them in the shopping cart 2, etc., to, in some cases, unauthorized actions or even fraudulent acts), and from that, it is possible to determine what action should be taken (fully or semi-automatically) in the data processing device 29.

[0129] The camera 40 can also be configured with software techniques to distinguish between events within its field of view that require an update of the digital twin and other events that barely require an update, which also limits the growing volume of image data to only that which is absolutely necessary, which is particularly advantageous with regard to the IoT functionality of the camera 40.

[0130] Finally, it should be further noted that the method 30 according to the invention can use only the rack rail camera 35, only the camera 40, or even a mixed configuration of the rack rail camera 35 and the camera 40 in connection with the updating step of the method. In particular, in a mixed configuration, the advantages of both cameras 35, 40 can be utilized or combined.

[0131] In this way, the use of the rack rail camera 35 provides optimal results when observing opposing racks 23 in the rack aisle, especially in narrow rack aisles, and is therefore even characterized by a deep view into the rack structure, which advantageously allows for updating the fixed parts of the sales area.

[0132] In contrast, the camera 40 is advantageous in locations where its position is essentially fixed, but where it can selectively access different parts of the sales area 27 by utilizing its rotating / swivel head 40, particularly with its sensor equipment and flexible optics, to update a digital image of the sales area 27. The use of a camera in these locations can be advantageous, even in locations where racks are installed but do not clearly define defined rack aisles, or where baskets of goods are installed or where goods are simply provided on tables, pallets, etc. From the fixed position of the camera, its flexible optics allow it to view a wide, three-dimensional area. The camera can, of course, also be installed above the rack aisles and serve to update the digital image of the rack aisles from there.

[0133] Finally, it should be pointed out once again that the figure detailed above is merely one example that can be modified in various ways by a person skilled in the art without departing from the scope of the invention. For the sake of completeness, it should also be pointed out that the use of an indefinite article in the singular does not exclude that the feature in question can also be present in the plural.

Claims

1. A method (30) for creating a digital image of a sales area (27), comprising the steps of the method listed below: a step (32) of computer-generating a digital image of the sales area (27) using recordings from at least one positionable camera device (3) providing a digital image of the surroundings of the path of travel within the sales area (27); a step (33) of computer-processing a portion of the digital image of the sales area (27) using recordings from at least one fixed-position camera device (35; 40) providing a digital image of the portion of the sales area (27); A method having the following.

2. 2. The method (30) of claim 1, The method wherein said updating step (33) occurs more frequently than the step (32) of generating digital images using a positionable camera device (3).

3. 3. The method (30) according to claim 1 or 2, The method is such that the fixed-position camera device (35) is a camera fixed to the product display equipment (23), in particular to the rack rail of the product display equipment (23), and the camera takes digital images of the part of the sales area (27) that is within the shooting range from its position according to its shooting range.

4. The method (30) according to any one of claims 1 to 3, The method is such that the fixed-position camera device (40) is a camera whose photographing direction is changeable, and this camera photographs digital images of the part of the sales area (27) that is within its photographing range according to the photographing range given by the photographing direction at each time from its position, and possibly while changing the photographing direction.

5. 5. The method (30) according to any one of claims 1 to 4, The method, wherein the updating step (33) comprises detecting the movement of objects, in particular during recordings of at least two fixed-position cameras (35; 40), and tracking such moving objects within a digital image of a portion of the sales area (27).

6. 6. The method (30) according to any one of claims 1 to 5, A method, wherein the updating step (33) is performed in a time-controlled manner, in particular periodically.

7. 7. The method (30) according to any one of claims 1 to 6, The method wherein the updating step (33) is automatically initiated depending on the situation.

8. The method (30) according to any one of claims 1 to 7, The method wherein the updating step (33) includes detecting an electronic display device (26).

9. 9. The method (30) according to any one of claims 1 to 8, The method, wherein the updating step (33) includes detecting a person.

10. 10. The method (30) according to any one of claims 1 to 9, The method wherein the updating step (33) includes detecting a product presence or absence state.

11. 11. The method (30) according to any one of claims 1 to 10, The method, wherein said updating step (33) is performed in real time upon the occurrence of at least one of the situations listed in claims 5 to 10.

12. 12. The method (30) according to any one of claims 1 to 11, The method wherein the photographing of the surroundings of the travel path is performed using a passive, position-variable camera device (3).

13. 13. The method of claim 12, The method in which the position-variable camera device (3) is a 360° camera, in particular a 360° camera fixed to a person's body or clothing, and this camera is used to take pictures in all directions around the movement path.

14. 14. The method (30) according to claim 12 or 13, The method in which the photographing of the surroundings of the travel path is performed using a position-variable camera device (3) attached to the shopping cart (2).

15. 15. The method (30) of claim 14, The method wherein the photographing of the surroundings of the travel path is performed from the basket (1) of the shopping cart (2).

16. 16. The method (30) of claim 15, The method wherein the photography of the surroundings of the travel path is carried out through openings in the basket (1) of the shopping cart (2) or in its walls, in particular through the bars (4) of the basket (1).

17. 17. The method (30) according to any one of claims 14 to 16, The method wherein said photographing of the surroundings of the travel path is performed from a position at the front of the shopping cart (2).

18. 18. The method (30) according to any one of claims 14 to 17, The method further comprises a camera device (3) for photographing the surroundings of the moving path, the camera device (3) having at least one photographing stage (8, 9, 10) equipped with a stereo camera.

19. 19. The method (30) according to any one of claims 14 to 18, The method, wherein the positionable camera device has at least a first and a second imaging stage, and advantageously also a third imaging stage (8, 9, 10), each imaging a separate spatial sector (20, 21, 22) around the path of movement.

20. 20. The method (30) of claim 19, The position-variable camera device (3) has three photographing stages (8, 9, 10) at the front of the shopping cart (2), and with respect to the front of the shopping cart (2), the first photographing stage (8) photographs a first spatial sector (20) diagonally forward to the left, the second photographing stage (9) photographs a second spatial sector (21) diagonally forward to the right, and the third photographing stage (10) photographs a third spatial sector (22) at the center front.

21. 21. The method (30) according to claim 19 or 20, The method wherein two of said imaging stages (8, 9, 10) are configured such that the spatial sectors (20, 21, 22) imaged by them overlap at least slightly.

22. 22. The method (30) according to any one of claims 1 to 21, said step (32) of generating a digital image of the sales area (27) comprises processing, by means of image processing software, in particular by means of software means technically called computer vision, the surroundings of the route of movement within the sales area (27) photographed by at least one positionable camera device (35; 40), As a result of this processing, a virtual space is realized as a digital image of the sales area, and this virtual space is visualized in a form that can be perceived by humans using visualization equipment, particularly a screen or equipment called a "Virtual Reality Headset" in English.

23. 23. The method (30) according to any one of claims 1 to 22, said step (33) of updating the digital image of the sales area (27) comprises processing the part of the sales area (27) photographed by at least one fixed-position camera device (35; 40) using image processing software, in particular software means known in the technical term as computer vision, A method in which a part of the digital image of the sales area (27) generated using this position-variable camera device (3) that corresponds to a part of the sales area (27) photographed using the fixed camera device (35; 40) is modified as a whole or in detail based on the recording of the fixed camera device (35; 40).

Images