Method and system for data transmission

EP4463952A1Pending Publication Date: 2024-11-20SIEMENS AG
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
EP2023707012
Authority / Receiving Office
EP · EP
Patent Type
Applications
Current Assignee / Owner
Priority Date
2022-02-16
Filing Date
2023-02-15
Publication Date
2024-11-20

AI Technical Summary

Technical Problem

Current diagnostic and maintenance interfaces for automation components, especially simpler devices like power supplies, face challenges in remote access, security, and hardware requirements, leading to inefficiencies and increased costs, particularly when dealing with complex automation systems.

Method used

A method utilizing a light emitter and camera-based system for data transmission, where images are captured sequentially as lines or columns, allowing for high-speed data communication and secure, local optical data transfer without the need for additional hardware, leveraging the 'rolling shutter' effect of CMOS sensors and signal processing techniques like Fast Fourier Transform and Goertzel algorithm to enhance data integrity and security.

Benefits of technology

This approach enables high-rate data communication with improved access protection and reduced susceptibility to failures, using a simple and robust optical interface that ensures secure, local data transfer and reduces hardware requirements, enhancing the reliability and security of automation system diagnostics.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 1.1
    Figure 1.1
Patent Text Reader

Abstract

Disclosed is a method for data transmission in a system (S), comprising a transmitter (TX) having a modulation device (MOD) and a light emitter, as well as a receiver (RX) having a camera, in which the image capture takes place sequentially in time in the form of image rows or image columns, and a computing device (CPU) with a memory, wherein the following steps are carried out: a) providing at least one piece of data information, b) modulating a carrier signal using the at least one piece of data information to form a modulated signal, c) transmitting the modulated signal, d) receiving the light signal (LS) through the camera, which captures an image (B), sequentially in time, in the form of image rows (Z1-Z8) or image columns (S1-S8), e) determining a first piece of image information (Bl1) and at least one second piece of image information (BI2-BI8) from the image (B) through row-wise or column-wise reading, f) forming a data stream by concatenating the first and the at least one second piece of image information (BI1, BI2-BI8), and g) calculating the signal frequency from the data stream and identifying the at least one piece of data information (DI) from the signal frequency.
Need to check novelty before this filing date? Find Prior Art

Description

[0001] Method and system for data transmission

[0002] The invention relates to a method for data transmission in a system comprising a transmitter with a modulation device and a light emitter, as well as a receiver with a camera in which the image capture takes place sequentially in the form of image lines or columns, and a computing device with a memory, wherein the following steps are carried out: a) providing at least one piece of data information, b) modulating a carrier signal with a carrier frequency using the at least one piece of data information to a modulated signal by the modulation device, c) transmitting the modulated signal through the light emitter of the transmitter as a light signal, d) receiving the light signal through the camera of the receiver, which captures an image, comprising image lines and image columns, sequentially in the form of image lines or columns, wherein an image line or an image column is captured at least twice as fast,as the period of the carrier signal is .,

[0003] The invention also relates to a system for data transmission, comprising a transmitter with a modulation device and a light emitter, as well as a receiver with a camera in which the image capture takes place sequentially in the form of image lines or image columns, and a computing device with a memory.

[0004] Automation components in electrical systems are becoming increasingly powerful and complex. For this reason, the need for inspection and maintenance is also increasing. While many devices already support remote maintenance and diagnostics, simpler components such as power supplies are typically not integrated into the automation network, yet play a critical role in the function of the overall system.

[0005] To enable digital maintenance and diagnostics for such devices, appropriate interfaces and diagnostic devices are required. The hardware and software requirements, both in the device itself and for the diagnostic device, are enormous, especially with a large number of such devices. Ideally, no additional hardware should be required for such an interface. Regarding the diagnostic device, no special hardware (dedicated diagnostic or readout devices) should be required either.

[0006] For components that are connected via the system network, remote diagnosis is generally possible. However, diagnosis and maintenance are often carried out on-site. If you want to diagnose a device, it must first be found in the automation system. Added to this are challenges with regard to connectivity in the field, identification of the device or access to the device (security), as well as restricting access to a device and isolating other participants in the automation system or communications network (i.e. unauthorized access by a service technician to other critical system components). A local interface would therefore also be advantageous here, even if the device is fundamentally integrated into the automation system.

[0007] Until now, diagnostics in automation components were carried out either via the plant network and the automation system or, in the case of simpler, non-connected devices, via local interfaces.

[0008] For this purpose, simple wired interfaces such as RS232 or RS485 were often created and communicated via simple proprietary protocols. In addition to the additional hardware required for the interface (including connectors and protection class), further challenges arise regarding interference-free operation, EMC, and conformity. In addition, this type of interface requires dedicated hardware for diagnostics (e.g., a handheld device or PC with a corresponding interface card).

[0009] Alternatively, wireless interfaces such as Bluetooth, Wi-Fi, or similar proprietary radio systems can be used. However, this requires a significant hardware upgrade, which in turn entails additional costs (hardware, EMC, compliance, etc.). When using proprietary systems, dedicated hardware for diagnostics is also required (e.g., a dedicated diagnostic device or PC interface). Radio systems also pose further challenges regarding access protection, security, and susceptibility to interference. In some industrial sectors, radio wave emissions are also a critical issue.

[0010] The object of the invention is to create a digital interface for a technical device which communicates data from the device to a receiver such as a mobile device in a simple manner, in particular with regard to access protection, security and susceptibility to interference.

[0011] The object according to the invention is achieved by a method of the type mentioned at the outset, wherein the following further steps are carried out: e) determining a first item of image information and at least one second item of image information from the image by reading them out line by line or column by column, analogously to their detection in step d), which are each based on at least a portion of the respective image lines or image columns, by the computing device, f) forming a data stream by concatenating the first and the at least one second item of image information by the computing device, g) calculating the signal frequency from the data stream and identifying the at least one item of data information from the signal frequency by the computing device.

[0012] This provides a simple way to communicate data at a high data rate from the device to a receiver such as a mobile device.

[0013] Improved access protection is also guaranteed, as the optical transmission means that data can only be received in the immediate vicinity and no intrusion into a connected IT network can result in the data being hacked by unauthorized persons, thus adversely affecting data security.

[0014] Furthermore, the solution is less susceptible to failure because the system is based on a very simple and robust principle.

[0015] The technical approach of the invention is based on the application of the properties of an electronic shutter together with CMOS sensors, as used in conventional cameras with electronic image sensors, such as in mobile phones, in particular for detecting communication in the optically visible range.

[0016] These peculiarities also manifest themselves in the usually undesirable "rolling shutter" effect in a camera of the type mentioned above.

[0017] The rolling shutter effect is a positioning error that can occur when taking photos or videos of moving objects. At first glance, it seems obvious that the exposure of all points on a camera's light-sensitive sensor element begins at exactly the same time, just as is the case with analog photography using a mechanical shutter.

[0018] However, there are cameras where, due to their design, this does not apply to the entire area. However, due to line-by-line or column-by-column reading, which occurs sequentially, the received light signal can change during the reading process of a line or column.

[0019] When all lines are combined to form a common image, the image would therefore change compared to the light signal which has changed in the meantime, so that the light signal cannot be displayed as a whole at once, but for example only line by line at different times.

[0020] The invention uses this effect to improve and increase the frame rate by capturing the light signal for each row- or column-wise readout process, thus producing a sensor data sequence sampled at a significantly higher time than for an entire image.

[0021] The rolling shutter effect occurs, for example, in digital cameras with image converters using CMOS sensor technology, since these sensors read out the image line by line or column by column.

[0022] In a further development of the invention, it is provided that during the determination, the first and the at least one second image information is obtained from the image by aggregating portions of the image rows or image columns.

[0023] This can further improve the camera's immunity to interference, as well as its optical sensitivity. In a further development of the invention, the signal frequency is calculated from the data stream using a fast Fourier transformation.

[0024] This allows the signal frequency to be determined in a reliable manner and the sensitivity to interference to be increased.

[0025] In addition, several temporally sequential modulation states can be detected simultaneously if the evaluation interval is chosen to be correspondingly long, i.e. if it extends over several bits in the at least one piece of data information.

[0026] In a further development of the invention, it is provided that the calculation of the signal frequency from the data stream is carried out using the Goertzel algorithm.

[0027] The Goertzel algorithm is a method from digital signal processing and represents a special form of the discrete Fourier transform (DFT). In contrast to the various fast calculation methods for the discrete Fourier transform (FFT), which always calculate all discrete spectral components in one block, the Goertzel algorithm makes it possible to calculate only individual discrete spectral components.

[0028] The algorithm is based on a structure consisting of a digital filter extended by a state controller. The states divide the calculation into a backward branch, into which the input values ​​sampled in the time domain are loaded, and a forward branch that provides the output signal. The backward loop is run through with each digital sample and is constructed as a recursive digital filter with two state memories and an accumulator. The forward branch is only run through once after N samples and provides the calculated complex output value from the state memories—namely, the spectral component in magnitude and phase.

[0029] The frequency selectivity can be adjusted by selecting the filter coefficients used. The quality factor can be influenced by selecting the number of samples. N can take any natural value.

[0030] However, a separate Goertz structure is required for each spectral component. Therefore, this algorithm is particularly advantageous and requires less computational effort when only individual spectral components are to be calculated, rather than the entire spectrum.

[0031] The Goertzel algorithm is particularly inexpensive and easy to implement. Furthermore, the computation time is shorter than in other prior art methods, especially when the target frequencies to be detected are known, as in the present method.

[0032] In a further development of the invention, it is provided that the modulation of the modulated signal is an amplitude, frequency and / or phase modulation, and the modulation frequency of the modulated signal is at least 50 Hz, preferably at least 500 Hz, particularly preferably at least 1000 Hz.

[0033] This allows the transmission rate of the data information to be significantly improved compared to an evaluation of the entire camera image.

[0034] In a further development of the invention, the modulation of the modulated signal is frequency modulation, and the modulated frequencies for individual modulation states are not harmonically located with each other. This simplifies the calculation of the signal frequency, as ambiguities that could adversely affect the evaluation are avoided.

[0035] In a further development of the invention, it is provided that the modulation of the modulated signal has a duty cycle of at least 50%, preferably at least 75%.

[0036] This allows a high light intensity to be achieved for the light emitter, which improves the range between transmitter and receiver as well as the susceptibility to interference and at the same time does not impair a signal function of the emitter, such as an operating status display.

[0037] In a further development of the invention, it is provided that the modulation of the modulated signal has a constant duty cycle over individual modulation states.

[0038] This allows the light emitter to ensure that the emitted light does not flicker for an observer, which improves information regarding the reliability of the technical device and does not impair the trust in the device for observers.

[0039] In particular , if the light emitter also serves as an operating indicator of the operating status of the technical device , flickering would be absolutely undesirable , as observers could mistakenly interpret this as a fault in the technical device

[0040] In a further development of the invention it is provided that the carrier frequency is constant.

[0041] This makes it possible to calculate the signal frequency from the data stream in a simple way and to increase the robustness of the system.

[0042] In a further development of the invention, it is provided that the at least one piece of data information comprises device-specific information, authorization information or authentication information.

[0043] This makes it possible, for example, for device-specific information, access information, authorization information or authentication information to be sent from the device via the optical interface to the receiver, for example in a smartphone.

[0044] A high level of security against eavesdropping is achieved because optical signal reception is only possible locally.

[0045] The object of the invention is also achieved by a system of the type mentioned at the outset, wherein the method steps according to the invention are carried out.

[0046] In a further development of the invention, it is provided that the transmitter is comprised by a technical device with an optical operating state indicator, and the light emitter is formed by the operating state indicator, and the receiver is comprised by a mobile device, in particular a smartphone or a tablet computer.

[0047] This allows a simple and cost-effective implementation of the system according to the invention.

[0048] The operating status indicator can be, for example, a light-emitting diode or elements of a digital display.

[0049] In a further development of the invention, it is provided that the at least one piece of data information comprises device-specific information, authorization information or authentication information of the technical device, which is provided for data transmission in the system.

[0050] Device-specific information, authorization information, or authentication information can be provided in the form of data information, sent by the sender, and received by the receiver. The invention is explained in more detail below using an exemplary embodiment illustrated in the accompanying drawings. The drawings show:

[0051] Fig. 1 is a block diagram of an embodiment of the invention,

[0052] Fig. 2 is a flow chart of the method according to the invention,

[0053] Fig. 3 An example of a modulated light signal,

[0054] Fig. 4 An example of a light signal received with a camera sensor according to Fig. 3,

[0055] Fig. 5 An example of a column-wise read-out signal of parts of the image according to Fig. 4,

[0056] Fig. 6 An example of an aggregated signal of Fig. 5,

[0057] Fig. 7 An example of a data stream formed from the aggregated signal according to Fig. 6.

[0058] Fig. 1 shows a block diagram for an embodiment of the invention.

[0059] A system S for data transmission comprises a technical device DEV, which has a transmitter TX with a modulation device MOD and a light emitter.

[0060] The light emitter may be an operating status indicator of the DEV device and may be, for example, a light-emitting diode or elements of a digital display of the DEV device.

[0061] Furthermore, the system S comprises a mobile radio device MD, a smartphone, with a receiver RX with a camera and a computing device CPU with a memory.

[0062] The computing device can be, for example, a microprocessor or a digital signal processor, a programmable logic device, or the like. In the camera, image capture occurs sequentially in the form of image lines Z 1 - Z 8 or, alternatively, in the form of image columns S 1 - S 8 , as further illustrated in Fig. 4.

[0063] A light signal LS is in the optically visible range and is transmitted from the emitter to the camera sensor.

[0064] The invention is based on a fundamentally disadvantageous feature of the camera and exploits the property of an "electronic shutter", which leads to a "rolling shutter" effect.

[0065] With the electronic shutter, the exposure of an entire image does not occur at exactly the same time during image capture.

[0066] The exposure and readout of the image information takes place line by line or column by column at a relatively high speed.

[0067] This allows signals to be transmitted through a sufficiently large light emitter, meaning the image section with the light emitter extends over several rows or columns. The decisive factors here are the line frequency or frequency and the exposure time of a row or column.

[0068] Fig . 2 shows a flow diagram of the method according to the invention, wherein the following steps are carried out: a) providing at least one piece of data information, b) modulating a carrier signal with a carrier frequency using the at least one piece of data information to a modulated signal by the modulation device MOD, c) transmitting the modulated signal through the light emitter of the transmitter TX as a light signal LS, d) receiving the light signal LS by the camera of the receiver RX, which captures an image B, comprising image lines Z1-Z8 and image columns S1-S8, sequentially in time in the form of image lines Z1-Z8 or image columns S1-S8, wherein an image line Z1-Z8 or an image column S1-S8 is captured at least twice as fast as the period TP of the carrier signal, e) determining a first piece of image information B11 and at least one second piece of image information BI2-BI8 from the image B by reading out line by line or column by column,analogous to their detection in step d), which are each based on at least a portion TM of the respective image lines Z1-Z8 or image columns S1-S8, by the computing device CPU, f) forming a data stream by concatenating the first and the at least one second image information B11, BI2-BI8 by the computing device CPU, g) calculating the signal frequency from the data stream and identifying the at least one data information DI from the signal frequency by the computing device CPU.,

[0069] The modulation of the modulated signal can be amplitude, frequency and / or phase modulation.

[0070] For a high data transmission rate, the modulation frequency of the modulated signal is at least 50 Hz or more, preferably at least 500 Hz, particularly preferably at least 1000 Hz.

[0071] If the modulation of the modulated signal is a frequency modulation, the modulated frequencies for individual modulation states of the modulation should not be harmonically located to each other.

[0072] For example, a frequency in the modulated signal for transmitting a logical "0" state can be 500 Hz, and a frequency in the modulated signal for transmitting a logical "1" state can be 700 Hz. Multi-valued modulations are also possible; for example, three different frequencies can be selected to transmit three-valued information, such as 1000 Hz, 1100 Hz, and 1200 Hz.

[0073] The modulation of the modulated signal has a duty cycle of at least 50% in order to achieve a high brightness of the LED.

[0074] With a higher duty cycle, the brightness of the LED is only slightly affected and is not or hardly noticeable to users of the DEV device.

[0075] Furthermore, the modulation of the modulated signal has a constant duty cycle across individual modulation states to prevent users of the DEV device from perceiving unwanted flickering.

[0076] To simplify the evaluation of the data stream, the carrier frequency is constant.

[0077] Fig. 3 shows an example of a modulated light signal with a period TP.

[0078] Fig. 4 shows an example of a light signal LS received by the camera sensor according to the previous figure.

[0079] The camera of the receiver RX captures an image B sequentially in time in the form of image columns S1-S8.

[0080] The image has image rows Z1-Z8 and image columns S1-S8.

[0081] An image column S1-S8 is acquired at least twice as fast as the period TP of the carrier signal.

[0082] For each column SI-S8, a subset TM is formed over several rows Z4, Z5.

[0083] Fig. 5 shows an example of a column-wise read-out signal of portions of the image according to Fig. 4. A first image information B11 and further image information BI2-BI8 are determined from the image B by column-wise readout of the camera.

[0084] Fig. 6 shows an example of an aggregated signal from Fig. 5.

[0085] During the determination, the first and second image information B11, BI2-BI8 can optionally be obtained from the image B by aggregating AG of portions TM of the image lines Z1-Z8 in order to improve the signal quality.

[0086] Alternatively, only one line from image B can be further processed.

[0087] Subsequently, a data stream is formed by concatenating the first and second image information B11, BI2-BI8 by the computing device CPU.

[0088] Fig. 7 shows an example of a data stream formed from the aggregated signal according to Fig. 6.

[0089] The signal frequency can be calculated from the data stream using a fast Fourier transform.

[0090] This can, for example, improve the signal quality of the light signal LS.

[0091] For example, ambient light, such as that produced by LED lighting, can be easily filtered out.

[0092] Lighting technology typically uses very similar frequencies in the range of 600-1000Hz.

[0093] If you apply an FFT or a similar method for extracting and detecting different frequencies to a section of a frame, you can clearly identify the fundamental frequency and thus the corresponding signal in the frequency domain. Depending on the size of the LED, the distance between the smartphone and the device (and thus the relative size of the LED in the image), as well as the line frequency, the EFT analysis will also produce significantly different values ​​for the different frequencies used for transmission.

[0094] The different parameters that influence the "measured" frequency, such as smartphone technology, size of the LED, etc., are irrelevant for detection, since only the relative frequency jumps between the bits in the data stream are relevant for decoding.

[0095] Despite different smartphone technologies, the frame rate and thus also the line frequency in normal video mode is comparable and sufficient for recognition.

[0096] Slow motion modes, which are implemented differently in many smartphones, are not required. In principle, the EFT can be calculated column by column across the image, so that each column contains a temporally and spatially offset series of pixels.

[0097] Values ​​of individual or selected FFTs can be summed, resulting in a single discrete spectrum and facilitating decoding.

[0098] Alternatively, for frequency measurement, time points TD1, TD2, TD3 for signal changes in the data stream can also be determined.

[0099] To do this, a start time TS can first be determined by providing the transmitter with a start sequence SI in addition to the data information DI, transmitting this sequence and receiving it with the receiver. If the start sequence SI is recognized, this signals that a data sequence will follow.

[0100] The signal frequency can be easily calculated from the times TD1, TD2, and TD3. A bit interval in the data stream spans several frames of the video recording, since decoding occurs asynchronously with encoding.

[0101] When using a simple UART transmission with a start bit, 8 data bits, a parity bit, and a frame rate of, for example, 60 fps (16.6 ms per frame), it is advantageous to have a bit interval of >50 ms, ensuring that at least two frames completely and correctly decode the bit, allowing majority voting to be used for decoding. However, a constant bit rate must be agreed upon in advance between the transmitter and receiver.

[0102] List of reference symbols:

[0103] 1-8 data sequence, aggregated image line portion across columns

[0104] 1A- 8A, 1B- 8B Image line ratio over columns

[0105] AE start sequence

[0106] AG Aggregation of image row share over columns

[0107] B Image

[0108] BI 1-BI 8 image information

[0109] CPU computing device

[0110] DEV device

[0111] DI Data Information

[0112] LS light signal, visible light

[0113] MD Mobile Device

[0114] MOD modulation device

[0115] RX receiver with camera

[0116] S System

[0117] S1-S 8 image columns

[0118] SI Start Information t Timeline

[0119] T1-T8 sampling times, readout times from camera

[0120] TD1-TD3 Time for change of data information

[0121] TM subset

[0122] TP Period of the light signal

[0123] TS start time

[0124] TX transmitter with light emitter

Claims

Patent claims 1. A method for data transmission in a system (S) comprising a transmitter (TX) with a modulation device (MOD) and a light emitter, and a receiver (RX) with a camera in which image acquisition is temporally sequential in the form of image lines or image columns, and a computing device (CPU) with a memory, wherein the following steps are performed: a) providing at least one piece of data information, b) modulating a carrier signal with a carrier frequency using the at least one piece of data information to form a modulated signal by the modulation device (MOD), c) transmitting the modulated signal by the light emitter of the transmitter (TX) as a light signal (LS), d) receiving the light signal (LS) by the camera of the receiver (RX), which captures an image (B) comprising image lines (Z1-Z8) and image columns (S1-S8) temporally sequentially in the form of image lines (Z1-Z8) or image columns (S1-S8).wherein an image line (Z1-Z8) or an image column (S1-S8) is acquired at least twice as fast as the period (TP) of the carrier signal, characterized by the following further steps: e) Determining a first image information (Bll) and at least a second image information (BI2-BI8) from the image (B) by reading out row by row or column by column, analogous to their acquisition in step d), each based on at least a portion (TM) of the respective image lines (Z1-Z8) or image columns (S1-S8), by the computing device (CPU) , f) Forming a data stream by concatenating the first and at least one second image information (Bll, BI2- BI8) by the computing device (CPU) , g) Calculating the signal frequency from the data stream and identifying the at least one data information (DI) from the signal frequency by the computing device (CPU) .

2. Method according to the preceding claim, wherein, in determining the first and at least one second image information (Bll, BI2-BI8) is obtained from the image (B) by aggregation (AG) of portions (TM) of the image lines (Z1-Z8) or image columns (S1-S8).

3. Method according to one of the preceding claims, wherein the calculation of the signal frequency from the data stream is performed using a Fast Fourier Transform.

4. Method according to one of claims 1 to 3, wherein the calculation of the signal frequency from the data stream is carried out using the Goert zel algorithm.

5. Method according to any of the preceding claims, wherein the modulation of the modulated signal is an amplitude, frequency and / or phase modulation, and the modulation frequency of the modulated signal is at least 50 Hz, preferably at least 500 Hz, particularly preferably at least 1000 Hz.

6. Method according to one of the preceding claims, wherein the modulation of the modulated signal is a frequency modulation and the modulated frequencies for individual modulation states of the modulation are not harmonically related to each other.

7. Method according to any of the preceding claims, wherein the modulation of the modulated signal has a duty cycle of at least 50%, preferably at least 75%.

8. Method according to any of the preceding claims, wherein the modulation of the modulated signal has a constant duty cycle over individual modulation states.

9. Method according to any of the preceding claims, wherein the carrier frequency is constant.

10. Method according to any of the preceding claims, wherein the at least one data information (DI) comprises device-specific information, authorization information or authentication information.

11. System (S) for data transmission, comprising a transmitter (TX) with a modulation device (MOD) and a light emitter, and a receiver (RX) with a camera in which the image acquisition takes place temporally sequentially in the form of image lines or image columns, and a computing device (CPU) with a memory, wherein the method steps are carried out according to one of the preceding claims.

12. System (S) according to the preceding claim, wherein the transmitter is comprised of a technical device with an optical operating status indicator, and the light emitter is formed by the operating status indicator, and the receiver (RX) is comprised of a mobile device (MP), in particular a smartphone or a tablet computer.

13. System (S) according to the preceding claim, wherein the at least one data information (DI) comprises device-specific information, authorization information or authentication information of the technical device which is intended for data transmission in the system (S).