DEVICE FOR RECORDING DATA TO PRODUCE A LOCATED STREET TRAIN PANORAMA AND METHOD FOR THIS

Description

[0001] The invention relates to a device for recording data to generate a georeferenced street panorama image using a camera and a satellite-based positioning and time-determination device. The device also includes a storage unit.

[0002] Furthermore, the invention relates to a method for recording data to generate a geolocated street panorama image.

[0003] The increasing population in urban areas leads to a rise in traffic, particularly private car traffic. Especially in city centers, this massive increase in traffic density causes problems resulting in a lack of space that can no longer be solved with simple measures. Further problems include the associated noise pollution from traffic and the increasingly critical burden of pollutants.

[0004] Studies have shown that approximately 30% of inner-city traffic is not classic transport traffic, but is due to vehicles looking for parking spaces.

[0005] Projections for Germany indicate that 1.9 billion hours are spent searching for parking spaces. This results in the consumption of 3.2 billion liters of fuel. The total economic loss is estimated at €40.4 billion.

[0006] The desire for parking management that reduces this traffic for parking is therefore increasingly prevalent.

[0007] Such systems fundamentally distinguish between two stages of data acquisition. Firstly, static data must be obtained, indicating the location of parking spaces, the permitted parking times, and how to park, for example, parallel or perpendicular parking. This also includes information on no-parking zones and any applicable parking fees. This type of data is referred to as static data and represents the basic requirement for parking management.

[0008] The second stage of data collection is the collection of so-called dynamic data, which provides information about the current parking space occupancy situation.

[0009] One method for determining static data is known, for example, from DE 10 2018 214 510 A1. However, this method can only provide information about whether a parking space is free or not. Data such as when parking is permitted or whether it is a driveway, for example, is difficult to ascertain.

[0010] On the one hand, Google Street View, for example, is known for generating georeferenced 360° images from various positions on a street using vehicles. However, such data is insufficient to generate enough information to calculate static parking data. For instance, the positions of individual permitted parking spaces cannot be determined at all or only with great difficulty. Similarly, data on permitted parking times and the like are often unavailable.

[0011] Another challenge lies in adding the corresponding location information to the generated images for a georeferenced street panorama. Creating individual images, similar to Google Street View, is usually straightforward. However, significant problems arise when images taken in close succession need to be generated continuously, as highly accurate location information is required.

[0012] The manual for a Garmin Virb camera ("Virb Series Owner's Manual", May 14, 2014 (2014-05-14), Found on the Internet: URL: http: / / static.garmin.com / pumac / VIRB_OM_EN.pdf) describes a device for recording data to create a geolocated street panorama image with a camera.

[0013] The invention therefore lies in the TaskThe basis is to specify a device and a method for recording data to generate a geolocated street panorama image which has a special location accuracy.

[0014] This problem is solved according to the invention by a device having the features of claim 1 and a method having the features of claim 9.

[0015] Advantageous embodiments of the invention are specified in the dependent claims, in the description, as well as in the figures and their explanation.

[0016] The device according to the invention comprises a processing unit configured to process time data from the satellite-based position and time determination system into a format recordable by the camera as coded time data and to forward this data to the camera for recording. Furthermore, the camera is configured to simultaneously record a continuous video and the coded time data as a video containing time data. The storage unit, in turn, is configured and configured to store the video containing the time data as well as the position and time data from the satellite-based position and time determination system.

[0017] The invention is based on several interconnected ideas. Firstly, it was recognized that capturing individual still images is insufficient or inefficient enough to obtain a sufficiently accurate panoramic image of a street, which can be used to determine static data for parking management. Therefore, according to the invention, a continuous video is generated. This video is recorded, for example, by a vehicle while driving along a street.

[0018] Furthermore, it was recognized that it is either impossible or only possible with disproportionate effort to link the satellite-based positioning and timing system's location data with the film. In particular, this would have to be done precisely to prevent any unwanted offset between the location data and the film's images.

[0019] To solve this problem, according to the invention, the processing unit converts the time or a time signal originating from the satellite-based positioning and time determination system into a format that the camera can record and forwards it to the camera. The camera records this time data as a processed time format, hereinafter referred to as coded time data, together with the actual image data. This results in the corresponding coded time data, which denotes a specific point in time, being precisely linked to the image at which the corresponding position data from the satellite-based positioning and time determination system was also determined.

[0020] Finally, according to the invention, the film with the coded time data and the position and time data originating from the satellite-based position and time determination device are stored on the storage unit.

[0021] In a subsequent evaluation, the exact position of each image in the film can be determined using the coded time data in the film and the time data which are linked to the position data.

[0022] The satellite-based positioning and time determination system can, for example, be a system based on the GPS, Galileo, Glonas and / or Beidou standard, or a combination thereof.

[0023] In a preferred embodiment, the processing unit is designed to generate the coded time data, process and output the coded time data as an acoustic signal, and forward it to the camera. In this way, a second channel, separate from the image data in a standard film, can be used to record the data. In principle, it would also be possible to feed an additional image signal into the camera, but this would require superimposing the external images to be recorded with the second image signal, which is not only complicated but would also result in a loss of quality in the recorded continuous film.

[0024] In principle, the time data can be encoded in any way. It is advantageous if the processing unit is configured to generate the encoded time data as a continuous time data stream, with one frame for each time value. In other words, data containing a time value is continuously sent to the camera. This time value is then recorded and stored along with the continuous video. By comparing the encoded time value recorded on the video with the corresponding parallel time value from the satellite-based positioning and time determination system, which is also stored on the memory unit, the exact position of each frame in the continuous video can be determined.

[0025] The continuous video is preferably recorded at 24 frames per second. However, higher frame rates, such as 25, 30, or even 60 frames per second, are also possible. The more frames captured, the higher the accuracy of the subsequently geolocated street panorama.

[0026] In principle, any encoding method can be used to generate the coded time data. However, it is preferred that the coded time data be generated in accordance with SMPTE-ITC. This is a time code introduced by the Society of Motion Picture and Television Engineers, which is used in television and studio production for video and audio synchronization. By default, the time coded in this way contains information about the hour, minute, second, and corresponding frame of the film, with this information depending on the recording standard used.

[0027] To achieve even greater accuracy, data in milliseconds and microseconds are inserted into the frame as a user bitload, in accordance with the invention. This means that, in addition to the aforementioned information about the hour, minute, second, and frame, information about the millisecond and microsecond is also present in each time data point.

[0028] To generate the time signal as the basis for this data, the processing unit can, for example, include a quartz crystal which is adjusted by a time signal from the satellite-based position and time determination system. This signal can, for example, be a second signal. As is known, quartz crystals are sensitive to temperature and exhibit drift, so it is necessary to repeatedly adjust the signal of the coded time data generated by the processing unit. For this purpose, a time signal from the satellite-based position and time determination system can be used, for example, according to the invention. The second signal is suitable for this purpose.

[0029] According to the invention, for example, the processing device is configured to abort and / or repeat one or more frames of the time data stream in order to adjust the coded time data using the time signal of the satellite-based position and time determination device.

[0030] In other words, if the processing unit, based on a time signal transmitted by the satellite-based positioning and timing device, detects that its encoded time signal is behind schedule, the generation of the time data stream is aborted and resumed with the correct information. However, if the processing unit detects that it is too fast, i.e., ahead of the time specified by the satellite-based positioning and timing device, then individual frames or parts of frames in the time data stream are repeated to realign with the correct time.

[0031] Higher data accuracy for creating the georeferenced street panorama can be achieved by additionally providing a LiDAR sensor unit with a LiDAR sensor. This unit can be configured to receive at least the time and / or position data from the satellite-based positioning and time determination system and, in turn, outputs the distance data determined by the LiDAR sensor along with the time and / or position data, which are stored in the memory unit.

[0032] In the subsequent creation of the georeferenced street panorama, further information about the exact distance from the camera to the nearest object is available. This provides an additional data basis.

[0033] Furthermore, the invention relates to a method for recording data to generate a georeferenced street panorama image using a camera and a satellite-based positioning and time-determination device. Additionally, a processing unit is provided which converts the time data from the satellite-based positioning and time-determination device into a format recognizable by the camera as encoded time data and forwards it to the camera. Simultaneously, the camera generates and outputs a continuous video containing the encoded time data. This video can then be stored in a corresponding device.

[0034] This method makes it possible to create a continuous video as a vehicle travels along a street and to store time data within it. If, in addition, time and position data from the satellite-based positioning and timing system are simultaneously recorded, these two data sets can later be used to determine the precise positions of the objects or images shown in the video when processing the georeferenced street panorama.

[0035] The encoded time data can be generated as an acoustic signal and recorded on the camera's audio track. This offers the advantage that no additional data, for example, needs to be superimposed on the camera's image. It is also relatively easy to use the audio track, as it is a standard feature on most cameras, although it is not required for recording the street scene.

[0036] The encoded time data can be generated as a continuous time stream, with one frame for each data point. This allows the current time to be recorded simultaneously on the camera, regardless of which frame the camera is currently capturing. This improves the subsequent determination of the position of the acquired data.

[0037] Preferably, the encoded time data is SMPTE-ITC compliant, with the option to add millisecond and microsecond data as a user bitload. This allows for not only the second or fractions of a second (depending on the number of frames used), as the smallest unit according to the LSMPTE-ITC standard, but also enables even higher accuracy via milliseconds and microseconds.

[0038] To further improve accuracy and, in particular, to prevent the coded time data generated by the processing unit from running too fast or too slow, the coded time data can be adjusted using a time signal, especially a second signal, from the satellite-based positioning and time determination system. Depending on whether, according to the processing unit, time has passed too quickly or too slowly, one or more frames of the time data stream can be aborted or repeated to restore synchronization with the time of the satellite-based positioning and time determination system.

[0039] Finally, a georeferenced street panorama can be generated from the continuous video containing the coded time data, using the position and time data from the satellite-based positioning and timing system. The exact recording position is determined for each frame of the continuous video using the coded time data. This is done by determining the time datum for each frame and then using the data also recorded by the satellite-based positioning and timing system to calculate the corresponding position. Because even microseconds are recorded, according to the encoding used, this allows for position determination down to a few centimeters.

[0040] The invention is explained in more detail below with reference to a schematic embodiment and the figures. These show: Fig. 1 a schematic view of a device according to the invention for recording data to generate a georeferenced street panorama image; Fig. 2 the basic structure of a time data stream according to the invention; Fig. 3 the principle of dropping a time data stream frame; Fig. 4 the principle of retransmitting a time data frame; and Fig. 5 a highly simplified street panorama image.

[0041] In Fig. 1 Figure 10 is a highly schematic, simplified representation of a device 10 according to the invention for recording data to generate a located street panorama image 60.

[0042] This device 10 includes a camera 12 and a satellite-based positioning and time determination device 14. The satellite-based positioning and time determination device 14 is, for example, a GPS receiver. The GPS signal transmits a highly accurate time signal via various satellites, from which the corresponding position can then be determined in the GPS device or GPS receiver.

[0043] Furthermore, the device 10 according to the invention includes a processing unit 18, which will be discussed in more detail below. Additionally, the device 10 according to the invention has a LIDAR sensor unit 22 with a LIDAR sensor 23. Both the satellite-based position and time determination device 14 and the camera 12 and the LIDAR sensor unit 22 are connected to a storage device 16.

[0044] The following section describes in more detail the functioning of the device 10 according to the invention. The aim is to use the device 10 to record data for generating a georeferenced street panorama 60. For this purpose, the device 10 is placed in a vehicle moving on a road. During the vehicle's journey, a video is recorded using the camera 12. In addition to the actual image data, position and time data are determined via the satellite-based positioning and timing device 14. These data are then synchronized with the video so that this information, or data, stored on the storage unit 16, can subsequently be processed to create a georeferenced street panorama 60.

[0045] For this purpose, at least one time signal is transmitted from the satellite-based position and time determination unit 14 to the processing unit 18. In the processing unit, the time signal is converted into a time data stream, which in turn is available as an audio stream. The exact structure of this audio stream will be described later with reference to the Figures 2, 3 and 4 received.

[0046] This audio stream is transmitted to camera 12 via an audio input located there. The audio stream contains highly accurate information about the current time. The camera then records its video, with the audio data present as an acoustic signal on the audio track. This video, along with the acoustic signal, is then stored on storage unit 16. For data protection reasons, this could be an encrypted storage device or a server that immediately encrypts the data so that it is not stored in raw form.

[0047] This makes it possible to determine and save the exact time of the creation of the film, that is, of an image of the film.

[0048] In parallel, the exact position and time are also transmitted from the device 14 to the storage unit 16 for satellite-based position and time determination and stored there.

[0049] Thus, in a subsequent evaluation, it is possible to determine the position at which each corresponding image of the film was created, using the exact time code on the film and the parallel time code of the position data.

[0050] In addition, the device 10 according to the invention, in the illustrated embodiment, includes the LIDAR sensor unit 22 with a LIDAR sensor 23. However, this is not strictly necessary.

[0051] In the embodiment shown here, at least time data from the device 14 for satellite-based position and time determination are also transmitted to the LIDAR sensor unit 22. The LIDAR sensor unit 22 processes this data together with the distance from the LIDAR sensor to the nearest object determined by the LIDAR sensor 23 and also stores this data using the storage unit 16.

[0052] In principle, it is also possible to transmit not only the time data from device 14 for satellite-based position and time determination, but also the position data. In this case, either only the position data with the corresponding distance data, or the time data, the position data, and the distance data would be processed by the LIDAR sensor unit 22 and stored in the storage unit 16. This data can then be used, analogous to the video data, to obtain further information for the street panorama image. It, too, is located with high precision, so that the exact position can be determined.

[0053] The following describes the simplified structure of the time data stream 30 with reference to Fig. 2This will be explained in more detail. Basically, the time data stream 30 represents an audio signal that encodes a bit sequence 37. The audio stream, or time data stream, is preferably designed to be SMPTE-ITC compliant. According to this standard, a specific point in time, i.e., a time data point, is encoded in each frame 31, 32, 33, 34, 35, 36. The in Fig. 2 The simplified time data stream shown, 30, contains only five frames per second. According to this standard, each frame encodes the current hour, minute, and second, as well as the current frame in relation to the recorded data. In a recording at 24 frames per second, this simplifies the information about which frame (31, 32, 33, 34, 35, 36) belongs to which image in the corresponding second.

[0054] According to the invention, this format was extended to include additional data on the millisecond and microsecond in the user bitload provided for in the standard.

[0055] The corresponding time date, which is encoded in each frame 31, 32, 33, 34, 35, 36, always corresponds to the beginning of the corresponding frame t1, t2, t3, t4, t5, t6. For simplification, it is assumed that only five images per second are generated, meaning that there are five frames in one second.

[0056] The processing unit 18 incorporates a quartz crystal to generate the highly accurate time signal that serves as the basis for the time data stream 30. Since quartz crystals and other timing devices are often temperature-dependent, it is necessary to adjust the time periodically. This is particularly important in the application presented here, as otherwise the highly accurate geolocation of the images cannot be achieved. For adjustment, the time signal from the satellite-based positioning and time determination unit 14 is used, for example. In this case, a second signal is used. This means that the satellite-based positioning and time determination unit 14 sends a signal to the processing unit 18 every second. The time data stream 30 is then readjusted using this signal.

[0057] The following will be discussed in Figs. 3 and 4The two cases described are those in which time on the processing unit 18 runs too slowly or too quickly.

[0058] Accordingly Fig. 3 The time calculated by the processing unit 18 is too slow. As a result, the second has already passed before the fifth frame 35 has finished. According to the invention, the transmission of this frame 35 is then immediately aborted, and the transmission of the next correct frame, i.e., frame 36, begins.

[0059] In another way, in Fig. 4This is shown when the processing unit 18 is running too slowly, meaning its timing is too slow. In this case, frame 36 starts before the next second has elapsed. This is also detected by the second signal transmitted by the satellite-based position and time determination unit 14. As a result, the transmission of frame 36 is again aborted, but then restarted accordingly, so that the time data stream 30 is once again synchronized with the time of the satellite-based position and time determination unit 14.

[0060] Finally, in Fig. 5A highly schematic example of a generated street panorama 60 is shown. During data processing, the continuous video data generated by the camera 20 is processed and assembled into a street panorama image 60 that exhibits as little or no optical distortion as possible. Individual objects 64, such as cars or trucks located in parking spaces, as well as signs 65, can then be identified on the street panorama image 60. These signs 65 may, for example, contain parking information. Complete or partial views of streetlights 66 are also shown. It is essential that the roadside is present to obtain sufficient data for static information for the parking management system. In the Fig. 5 The hatched objects are, for example, houses 67 located in the background.

[0061] With the device according to the inventive method, it is therefore possible to efficiently generate data which can be used to create a geolocated street panorama image.

Claims

1. Apparatus (10) for recording data to produce a localized panoramic image of a street with a camera (12), with a device (14) for satellite-based position and time determination, with a storage unit (16), characterized in that a preparation device (18) is provided which is designed to prepare time data of the device (14) for satellite-based position and time determination into a format recordable for the camera (12) as coded time data as an acoustic signal and to forward this to the camera (12) for recording, in that the camera (12) is designed to simultaneously record a continuous figuratively film and the coded time data as an acoustic signal as a film with time data, in that the storage unit (16) is designed and configured to store the film with coded time data and in that the storage unit (16) is designed and configured to store position and time data of the device (14) for satellite-based position and time determination, in that the preparation device (18) is set up to generate the coded time data in a SMPTE Itc compliant manner whilst inserting data on the millisecond and microsecond as user bit load.

2. Apparatus (10) according to claim 1, characterized in that the preparation device (18) is configured to produce the coded time data as a continuous time data stream with a frame for each time-stamp.

3. Apparatus (10) according to any one of claims 1 or 2, characterized in that the preparation device (18) is configured to adjust the coded time data by means of a time signal, in particular a seconds signal, of the device (14) for satellite-based position and time determination.

4. Apparatus (10) according to any one of claims 2 to 3, characterized in that for adjustment of the coded time data the preparation device (18) is configured to stop and / or repeat one or several frames in order to adjust the coded time data by means of the time signal of the device (14) for satellite-based position and time determination.

5. Apparatus (10) according to any one of claims 1 to 4, characterized in that a LIDAR sensor unit (22) with a LIDAR sensor (23) is provided which is designed to receive position and / or time data from the device (14) for satellite-based position and time determination and which is configured to transmit distance data of the LIDAR sensor (23) together with position and time data of the device (14) for satellite-based position and time determination.

6. Method for recording data to produce a localized panoramic image of a street with a camera (12), with a device (14) for satellite-based position and time determination and with a preparation device (18), wherein the preparation device prepares time data of the device (14) for satellite-based position and time determination into a format recordable for the camera (12) as an acoustic signal as coded time data and forwards this to the camera (12), wherein the camera (12) produces and issues a continuous figuratively film simultaneously with the coded time data as an acoustic signal as a film with time data, in that the coded time data are SMPTE Itc-compliant and in that as user bit load data on the millisecond and microsecond are inserted.

7. Method according to claim 6, characterized in that the coded time data are produced as a continuous time data stream with a frame for one time-stamp in each case.

8. Method according to any one of claims 6or7, characterized in that the coded time data are adjusted by means of a time signal, in particular a seconds signal, of the device (14) for satellite-based position and time determination.

9. Method according to any one of claims 6 to 8, characterized in that for adjustment of the coded time data one or several frames are stopped and / or repeated.

10. Method according to any one of claims 6 to 9, characterized in that a localized panoramic image of a street is produced from the continuous film with the coded time data whilst using the position and time data of the device (14) for satellite-based position and time determination.