Vehicle-based quick response code scanning method

Abstract

The present invention relates to a vehicle-based quick response code scanning method. A method for scanning a quick response (QR) code, the method executable by a processor on a vehicle during a vehicle-merchant transaction, the method comprising, while a QR code is displayed on a display screen of an infrastructure node, using a camera mounted to the vehicle to collect camera data of the QR code as the vehicle moves relative to the display screen. The method comprises transmitting the camera data to a processor of a camera control system, processing the camera data via the processor of the camera control system to determine characteristics of the camera data, and performing a multi-factor health check of the QR code. The method also comprises, in response to one or more of the values of the characteristics being less than a corresponding threshold value of the multi-factor health check, performing a control action of the camera and / or the vehicle.

Description

Technical Field

[0001] This disclosure relates to an automated method for optimizing Quick Response (QR) code scanning in a vehicle using one or more vehicle cameras. A QR code is a two-dimensional barcode that stores information such as website addresses, plain text, contact information, menus, etc. The information is first converted into binary data and then encoded into a pixelated black-and-white pattern. In a typical QR code, three large squares are positioned at the corners to allow the QR code to be identified and properly oriented by a QR code scanner (e.g., a smartphone camera, or, in this application, a vehicle camera). Smaller squares are used to help correct for distortion. The QR code also includes version information corresponding to the module size / data capacity of the QR code, as well as data and an error correction key that form the encoded information and associated error correction code. Background Technology

[0002] To scan a QR code, the scanning device's camera detects and views the QR code, which can then be displayed on a screen or printed in a tangible form on a surface such as a sticker, product, sign, or menu. The scanning device's image processing software locates the QR code within the camera's field of view and orients it before decoding the aforementioned binary data. If necessary, error correction can be applied before decoding. Depending on the nature of the encoded data, the scanning device can respond to successful decoding of the information by performing or requesting one or more actions, such as, but not limited to, opening a website, displaying text, adding contact information, or performing countless other possible actions. Summary of the Invention

[0003] This document discloses a method and system for obtaining accurate Quick Response (QR) code scanning results using one or more cameras on a vehicle when QR codes are displayed at roadside booths, toll booths, drive-thru merchants, or other infrastructure nodes. This solution achieves successful QR code scanning through camera parameter tuning and real-time QR code "health checks." To obtain healthy QR codes, the method described herein makes decisions about whether a change in vehicle location is necessary. Using optional crowdsourced QR code locations, the vehicle's camera control system is alerted to a better or more feasible location for proper QR code reading. Among other potential benefits, this strategy minimizes premature fault alerts and unnecessary user action requests.

[0004] In a representative scenario, a display screen operable for displaying QR codes is connected to or juxtaposed with the aforementioned infrastructure node. In traditional QR code-based transactions, the user adjusts the camera (e.g., a smartphone camera) so that it is positioned near and directly facing the QR code. This action enables the camera and associated software to correctly scan and decode the displayed QR code. However, this paper envisions a use case where the vehicle and its(one or more) onboard cameras move relative to the display screen / infrastructure node. During such a scenario, an important consideration is whether the QR code is adequately positioned at the current location of the vehicle and camera, or whether it is prudent to wait and / or change the vehicle / camera position before rescanning. Using this teaching, an onboard camera control system can address this consideration by selectively adjusting camera settings, utilizing crowdsourcing, and leveraging the aforementioned QR code health check functions (e.g., skewness and distance) to ensure proper vehicle placement for optimal decoding of the QR code information.

[0005] Specifically, this document discloses a method for scanning QR codes during vehicle-to-merchant transactions. Embodiments of this method include collecting camera data of the QR code using a camera mounted on the vehicle while the vehicle moves relative to the display screen, when the QR code is displayed on a screen at an infrastructure node. The method may also include transmitting the camera data to a processor of a camera control system, processing the camera data via the processor to determine characteristics of the camera data, and performing a multi-factor health check on the QR code. The method also includes performing control actions on the camera and / or the vehicle in response to one or more values ​​of the characteristics being less than a corresponding threshold of the multi-factor health check.

[0006] Factors in a multi-factor health check can include the sharpness of image data; in this case, control actions can include adjusting camera settings to optimize QR code scanning. Adjusting camera settings can include adjusting one or more of sharpness, brightness, or size settings. Adjusting sharpness or brightness settings can include adjusting the camera's shutter speed, while adjusting size settings can include adjusting the camera's zoom level.

[0007] In one or more embodiments, the factors in the multi-factor health check include the viewing angle of the QR code. Control actions may include adjusting the camera's pan and / or tilt settings when the viewing angle exceeds a predetermined viewing angle as a corresponding threshold. Adjusting the camera's pan and / or tilt settings may include transmitting motor control signals to the camera's motors via a processor.

[0008] In possible implementations, performing control actions includes causing a change in the vehicle's position relative to the QR code when the viewing angle exceeds a viewing angle threshold. For example, the vehicle may be configured as an autonomous vehicle, and causing a change in the vehicle's position relative to the QR code may include controlling the autonomous vehicle's steering and / or propulsion states via a processor. Alternatively, causing a change in the vehicle's position relative to the QR code may include prompting the vehicle's driver to change the vehicle's steering and / or propulsion states via a human-machine interface.

[0009] The method may include receiving crowdsourced data indicating the previously scanned location of a QR code, processing the crowdsourced data via a processor of a camera control system to determine an optimal placement area for the vehicle, and controlling the state of the vehicle to guide it to the optimal placement area. Embodiments of the method include determining a QR code version via the processor, calculating a pixel count per module based on the QR code version, and decoding the QR code via the processor when the pixel count per module exceeds a predetermined pixel count per module threshold.

[0010] This document also discloses a camera control system operable for scanning QR codes during vehicle-to-business transactions. The camera control system includes an onboard camera, a processor, and a computer storage medium (“memory”) on which instructions are recorded. When a QR code is displayed on a display screen at an infrastructure node and the vehicle moves relative to the display screen, the processor executes instructions to cause the camera control system to collect camera data of the QR code via the onboard camera, process the camera data to determine characteristics of the camera data, and perform a multi-factor health check on the QR code. The execution of the instructions also causes the processor to perform control actions on the camera and / or the vehicle in response to one or more values ​​of the characteristics being less than a corresponding threshold of the multi-factor health check.

[0011] This invention includes the following technical solutions:

[0012] 1. A method for scanning a Quick Response (QR) code during a vehicle-to-merchant transaction, comprising:

[0013] When the QR code is displayed on the display screen of the infrastructure node, as the vehicle moves relative to the display screen, camera data of the QR code is collected using a camera mounted on the vehicle;

[0014] The camera data is transmitted to the processor of the camera control system;

[0015] The camera data is processed by the processor of the camera control system to determine the characteristics of the camera data;

[0016] Perform the multi-factor health check using the QR code; and

[0017] In response to one or more of the characteristics having a value less than the corresponding threshold of the multifactor health check, control actions are performed on the camera and / or the vehicle.

[0018] 2. The method according to Scheme 1, wherein the factors of the multi-factor health check include the clarity of the image data, and wherein the control action includes adjusting the camera settings to optimize the scanning of the QR code.

[0019] 3. The method according to Scheme 2, wherein adjusting the camera settings includes adjusting one or more of sharpness settings, brightness settings, or size settings.

[0020] 4. The method according to Scheme 3 includes adjusting the sharpness setting or the brightness setting, wherein performing the control action includes adjusting the shutter speed of the camera.

[0021] 5. The method according to Scheme 3, including adjusting the size setting, wherein performing the control action includes adjusting the zoom level of the camera.

[0022] 6. The method according to Scheme 2, wherein the factors of the multi-factor health check include the viewing angle of the QR code, and wherein the control action includes adjusting the panning setting and / or tilt setting of the camera when the viewing angle exceeds a predetermined viewing angle as the corresponding threshold.

[0023] 7. The method according to claim 6, wherein adjusting the panning setting and / or tilt setting of the camera includes transmitting motor control signals to the camera motor via the processor.

[0024] 8. The method according to Scheme 6, wherein performing the control action includes causing a change in the position of the vehicle relative to the QR code when the viewing angle exceeds a viewing angle threshold.

[0025] 9. The method according to claim 8, wherein the vehicle is an autonomous vehicle, and wherein the change causing the position of the vehicle relative to the QR code includes controlling the steering and / or propulsion state of the autonomous vehicle via the processor.

[0026] 10. The method according to claim 8, wherein the change in the position of the vehicle relative to the QR code includes prompting the driver of the vehicle to change the steering and / or propulsion state of the vehicle via a human-machine interface.

[0027] 11. The method according to Scheme 1 further includes:

[0028] Receive crowdsourced data indicating the previous scan location of the QR code;

[0029] The crowdsourced data is processed by the processor of the camera control system to determine the optimal placement area for the vehicle; and

[0030] Control the state of the vehicle so that it is guided to the optimal placement area.

[0031] 12. The method according to Scheme 1 further includes:

[0032] The processor determines the QR code version.

[0033] Calculate the pixel count for each module based on the QR code version; and

[0034] When the pixel count per module exceeds a predetermined pixel count threshold per module, the QR code is decoded by the processor.

[0035] 13. A camera control system for scanning Quick Response (QR) codes during vehicle-to-merchant transactions, comprising:

[0036] Car-mounted camera;

[0037] Processor; and

[0038] A computer storage medium (“memory”) having instructions stored thereon, which can be executed by the processor to cause the camera control system to perform the following operations:

[0039] When the QR code is displayed on the display screen of the infrastructure node and the vehicle moves relative to the display screen, camera data of the QR code is collected via the vehicle-mounted camera;

[0040] The camera data is processed by the processor to determine the characteristics of the camera data;

[0041] Perform the multi-factor health check using the QR code; and

[0042] In response to one or more of the characteristics having a value less than the corresponding threshold of the multifactor health check, control actions are performed on the camera and / or the vehicle.

[0043] 14. The camera control system according to claim 13, wherein the factors of the multi-factor health check include the sharpness of the image data, and wherein the instruction is executable by the processor to cause the camera control system to adjust the camera settings as the control action to optimize the scanning of the QR code, the settings including sharpness settings, brightness settings, or size settings.

[0044] 15. The camera control system according to claim 13, wherein the factors of the multi-factor health check include the viewing angle of the QR code, and wherein the instruction is executable by the processor to cause the camera control system to adjust the panning setting and / or tilt setting of the camera as the control action when the viewing angle exceeds a predetermined viewing angle as the corresponding threshold.

[0045] 16. The camera control system according to claim 13, wherein the instructions are executable by the processor to cause the camera control system to act as a control system, causing a change in the position of the vehicle relative to the QR code when the viewing angle exceeds a viewing angle threshold, including causing the camera control system to request a change in the steering and / or propulsion state of the vehicle via the processor.

[0046] 17. The camera control system according to claim 16 further includes:

[0047] A human-machine interface (HMI), wherein the instructions are executable by the processor to cause the camera control system, as the control system, to prompt the driver of the vehicle to change the steering and / or propulsion state of the vehicle via the HMI.

[0048] 18. A vehicle comprising:

[0049] Vehicle body;

[0050] The wheel assembly connected to the vehicle body; and

[0051] A camera control system operable to scan Quick Response (QR) codes during vehicle-to-merchant transactions between the vehicle and infrastructure nodes, the camera control system comprising:

[0052] Car-mounted camera;

[0053] Processor; and

[0054] A computer storage medium (“memory”) having instructions stored thereon, which can be executed by the processor to cause the camera control system to perform the following operations:

[0055] When the QR code is displayed on the display screen of the infrastructure node and the vehicle moves relative to the display screen, camera data of the QR code is collected via the vehicle-mounted camera;

[0056] The camera data is processed by the processor to determine the characteristics of the camera data;

[0057] Perform a multi-factor health check on the QR code, including (i) checking the clarity of the image data and (ii) checking the viewing angle of the QR code;

[0058] In response to one or more of the characteristics being less than the corresponding threshold of the multifactor health check, the camera control action is performed by adjusting one or more settings of the camera to optimize the scanning of the QR code.

[0059] 19. The vehicle according to claim 18, wherein the vehicle-mounted camera includes a camera motor operable to control the panning level and / or tilt level of the vehicle-mounted camera, and wherein the instructions are executable by the processor to cause the camera control system to adjust the panning setting and / or tilt setting of the camera via the camera motor when the viewing angle exceeds a predetermined viewing angle.

[0060] 20. The vehicle according to claim 18, wherein the vehicle is configured as an autonomous vehicle, and wherein the instructions are executable by the processor to cause an autonomously controlled change in the position of the vehicle relative to the QR code when the viewing angle exceeds a viewing angle threshold.

[0061] The foregoing and other features and advantages of this disclosure will become apparent when taken in conjunction with the accompanying drawings and the appended claims, based on the following detailed description of illustrative examples and models for carrying out this disclosure. Furthermore, this disclosure explicitly includes combinations and sub-combinations of the elements and features presented above and below. Attached Figure Description

[0062] Figure 1 This is an illustration of an exemplary operational scenario in which a moving vehicle reads a displayed Quick Response (QR) code from a display screen on an infrastructure node, aided by programming capabilities of an onboard camera control system.

[0063] Figure 2 The diagram shows the possible connections. Figure 1 An exemplary camera control system used in vehicles.

[0064] Figure 3 This is a flowchart describing a method for QR code discovery as an aspect of this strategy.

[0065] Figure 4 This is a flowchart describing the method for optimal vehicle placement when scanning the displayed QR code using an onboard camera.

[0066] Figure 5 and Figure 6 The illustration shows the aspects of the displayed QR code considered by the camera control system during the execution of this method.

[0067] Figure 7A , Figure 7B and Figure 7CThe illustrations depict the relative long, medium, and short distances between the vehicle-mounted camera and the displayed QR code according to aspects of this disclosure.

[0068] Figure 8 This is a flowchart describing a method for vehicle placement as another aspect of this strategy.

[0069] Figure 9 It describes the use of tuning Figure 1 A flowchart outlining the various aspects of setting up a vehicle-mounted camera.

[0070] This disclosure may be modified or embodied in alternative forms, wherein representative embodiments are shown in the accompanying drawings and described in detail below. The inventive step of this disclosure is not limited to the disclosed embodiments. Rather, this disclosure is intended to cover alternatives that fall within the scope of this disclosure as defined by the appended claims. Detailed Implementation

[0071] Referring to the accompanying drawings, where similar reference numerals are used throughout several views to indicate similar features, Figure 1 The diagram illustrates a network architecture 10 with infrastructure node 11 and vehicle 12, which is shown traveling along road surface 13 in the direction of arrow AA. Traveling in the indicated direction will ultimately place vehicle 12 within the optimal placement area 20 relative to infrastructure node 11. The determination of the optimal placement area 20 and other decisions are based on the following references. Figures 2-9 The contents of this disclosure are as set forth.

[0072] Vehicle 12 may include a body 14 and a set of wheels 18. In the vehicle-merchant transaction considered herein, vehicle 12 moves relative to infrastructure node 11 (e.g., roadside booth, toll booth, shop, or another drive-by structure acting as a point of sale / transaction point). Within the scope of this disclosure, vehicle 12 includes one or more onboard cameras 16. Each camera 16 is securely mounted to the body 14, for example, to a rearview mirror, dashboard, windshield, or other forward-facing location. In some embodiments, camera 16 may also be equipped with camera motor 160 (e.g., a motorized gimbal system and / or (one or more) servo motors), wherein in one or more embodiments, camera motor 160 is responsive, for example, to a motor control signal (CC). 160 Provides vertical and horizontal axis control for camera 16 (see...) Figure 2 ).

[0073] During the propulsion mode of vehicle 12, one or more wheels 18 may be driven by torque from an internal combustion engine, an electric traction motor, or other prime mover (not shown). In such an embodiment, the powered rotation of the wheels 18 propels vehicle 12 toward infrastructure node 11 and a display screen 22 connected to or juxtaposed with infrastructure node 11. Although for illustrative purposes, vehicle 12 in Figure 1 While depicted as a passenger vehicle, other types of vehicles and other mobile platforms, such as trucks, boats, motorcycles, bicycles, agricultural equipment, etc., can be used within the scope of this disclosure without limitation. Particularly for marine applications, the road surface 13 will be replaced by a body of water, where infrastructure nodes 11 (e.g., roadside businesses as shown) may be represented as kiosks, rafts, etc., at a dock. For the sake of consistency, vehicle 12 will be described below with respect to its representative construction as a passenger motor vehicle, without limiting its construction to such an embodiment.

[0074] In possible use cases, Figure 1 The driver / operator of vehicle 12 may wish to transact with potentially distant merchants or service providers via dynamic and instantaneous interaction with infrastructure node 11. For example, the operator may wish to purchase goods or services, or may gain access to toll roads, bridges, parking garages, or other restricted access locations. For this purpose, onboard camera 16 (described hereinafter as singular for consistency only) can be automatically focused on a display screen 22 located at a distance in front of vehicle 12. As described above, display screen 22 can be securely connected to and / or juxtaposed with infrastructure node 11.

[0075] As part of the vehicle-to-merchant transaction considered herein, a merchant back office (not shown), server, or other goods or service provider may display a Quick Response (QR) code 24 on display screen 22. To complete the transaction, camera 16 is operable to detect the QR code 24 within its field of view / camera focus area, zoom in on the QR code 24, and subsequently scan / read multiple frames of the QR code 24. However, the distance and movement of vehicle 12 relative to infrastructure node 11 and display screen 22 can complicate the scanning process and reduce its accuracy, potentially requiring several scanning attempts or "re-scanning." Therefore, this teaching is provided as a solution embodied in computer-readable instructions executed by camera control system 25 of vehicle 12 to improve the effectiveness and accuracy of QR code-based vehicle-to-merchant transactions performed by vehicle 12 and infrastructure node 11.

[0076] refer to Figure 2In a possible implementation, the camera control system 25 includes one or more processors (P) 26 and a non-transitory computer-readable storage medium (“memory”) (M) 27. The camera (CAM) 16 (e.g., a camera based on a high-resolution complementary metal-oxide-semiconductor (CMOS) sensor) may, in some configurations, use a camera motor 160 for panning or tilting, the camera motor 160 being responsive to a motor control signal (CC). 160 For example, to adjust the pan or tilt settings of camera 16. Camera 16 transmits data in the form of camera data (CC) via, for example, hardwired or wireless connection. 16 The digital image is transmitted to the processor 26. The memory 27 includes non-transitory storage media / devices (read-only, programmable read-only, solid-state, random access, optical, magnetic, etc.). The memory 27 is capable of storing instructions in the form of one or more software or firmware programs or routines, one or more combinational logic circuits, one or more input / output circuits and devices, signal conditioning and buffering circuits, and other components accessible by one or more processors to provide the described functions. On this memory 27, computer-readable instructions embody the method 100 described below. Figure 3 ) and various sub-methods 200, 300 and 400 (respectively) Figure 4 , Figure 8 and Figure 9 ).

[0077] Additionally, regarding the camera control system 25, the input / output circuitry and devices include analog-to-digital converters and related devices for monitoring inputs from sensors, wherein such inputs are monitored at a preset sampling frequency or in response to a trigger event. Software, firmware, programs, instructions, control routines, code, algorithms, and similar terms refer to a set of instructions executable by the controller, including calibration and lookup tables. The camera control system 25 executes one or more control routines to provide the desired functionality. Ultimately, the camera control system 25 outputs parameter control signals (arrow CC). O ( ), to adjust one or more parameters or functions of the vehicle-mounted camera 16.

[0078] To achieve this solution, Figure 1 Infrastructure node 11 may be equipped with a sensor suite 26 operable to detect the relative position between vehicle 12 and display screen 22 on which a QR code 24 is ultimately displayed. The resident sensors of sensor suite 26 may include (by way of example and not limitation) infrastructure cameras, lidar sensors, radar sensors, ultra-wideband (UWB) sensors, etc. To facilitate onboard processing via camera control system 25, vehicle 12 is configured to use, for example, 5G cellular, WiFi connectivity, and Bluetooth. TMBluetooth Low Energy (BLE) or other suitable wireless communication protocols, such as IEEE 802.11, WiMAX, etc.

[0079] like Figure 2 As schematically illustrated, the Human-Machine Interface (HMI) 29 can be used as part of the camera control system 25, or it can be positioned to communicate with the processor 26 of the camera control system 25. For example, a vehicle infotainment system (not shown) can display icons or text prompts to the driver of vehicle 12 regarding pre-QR code scanning activities. If vehicle 12 is not in the optimal position but will soon be in the optimal position, the processor 26 can cause HMI 29 to display a "Search" icon (e.g., a magnifying glass or other intuitive icon), or a "System Processing" icon or text message. If vehicle 12 has missed the optimal position and driver intervention becomes necessary, HMI 29 can remind the driver of this fact via appropriate icons, text or other markings, possibly coupled with icons for a handheld smartphone to increase clarity, such as "Manual Scan Required".

[0080] when Figure 1 When vehicle 12 moves relative to infrastructure node 11, and camera 16 attempts to scan / read a QR code 24 located in front of vehicle 12 and possibly displayed at an angle relative to vehicle 12, several factors are considered to help optimize the scanning task. For example, while camera 16 may be able to perceive aspects of the QR code 24 at a great distance, that distance may be too great and / or the approach angle of the QR code 24 may be too significant, causing camera 16 to fail to perceive and decode it accurately. Similarly, depending on the distance / approach angle, the resolution of camera 16 may be too low. Although the QR code has embedded a special pattern to help reduce distortion and align with the pattern, problems may still arise when attempting to view and scan the QR code 24 from an angle. For example, pixel resolution decreases significantly at greater distances / angles, leading to QR code reading failures.

[0081] In order to solve Figure 1 These and other potential performance issues in vehicle-to-merchant transaction scenarios Figure 2 The camera control system 25 is configured to achieve successful scanning of the QR code 24 via selective tuning of the parameters of the camera 16 and real-time health checks of the generated QR code 24. Additionally, regarding optional crowdsourced locations for scanning the QR code 24, the camera control system 25 can identify better / more feasible locations (e.g., ...). Figure 1The optimal placement area 20 is used to accurately obtain the QR code reading. In such an embodiment, the method 100 described below may include receiving crowdsourced data indicating the previous scan position of the QR code 24, processing the crowdsourced data via the processor 26 of the camera control system 25 to determine the optimal placement area 20 of the vehicle 12, and ultimately controlling the state of the vehicle 12 so that the vehicle 12 is autonomously or manually guided to the optimal placement area 20.

[0082] refer to Figure 3 This illustrates a method 100 for scanning a QR code 24 during a vehicle-merchant transaction. The execution of method 100 enables... Figure 1 The camera control system 25 of the vehicle 12 shown is able to accurately detect the QR code 24. Method 100 can be embodied in computer-readable instructions and is recorded in... Figure 2 The camera control system 25 shown is stored in memory 27. For clarity, method 100 is described in segments or blocks of code, each executable by processor 26 to cause the camera control system 25 to perform the described functions. Boxes B102-B106 collectively describe the initial work performed by the camera control system 25, while the remaining boxes B108-B116 perform actions based on results, sharpness, skewness, and other factors.

[0083] Generally speaking, by Figure 2 The camera control system 25 execution method 100 includes when Figure 1 When QR code 24 is displayed on display screen 22 of infrastructure node 11, camera data (CC) of QR code 24 is collected. 16 This action is performed using camera 16 when vehicle 12 moves relative to display screen 22 / infrastructure node 11. Figure 1 The processor 26 of the camera control system 25 shown processes camera data (CC). 16 ) to determine the camera data (CC) 16 The camera control system 25 also performs a multi-factor health check on the QR code 24, and in response to one or more values ​​of the characteristics being less than the corresponding threshold of the multi-factor health check, performs control actions on the camera 16 and / or the vehicle 12.

[0084] The aforementioned health check results in one of two actions: (1) when the collected camera data (CC) 16 When the image is not sharp enough, the camera control system 25 can automatically adjust one or more camera settings of the camera 16, for example, using... Figure 9 Method 400. If the center line (LL) of the displayed QR code 24 is severely skewed relative to the threshold (e.g. Figure 5 and Figure 6(As shown in the diagram), the camera control system 25 can alternatively tune the vehicle 12 relative to... Figure 1 The position of the display screen 22. When the vehicle 12 is configured as an autonomous vehicle 12 (e.g., fully autonomous or self-driving), this can occur autonomously, such as by, for example, when the viewing angle exceeds a viewing angle threshold, via Figure 2 The processor 26 autonomously controls the steering and / or propulsion states of the autonomous vehicle. Alternatively, method 100 may include controlling the steering and / or propulsion states of the autonomous vehicle via... Figure 2 HMI 29 prompts the driver of vehicle 12 to change the steering and / or propulsion status of vehicle 12, thereby changing the position of vehicle 12 relative to QR code 24. This action essentially helps vehicle 12 or other mobile host systems gradually enter... Figure 1 The optimal placement area is 20, as shown below. Figure 4 , Figures 7A-7C and Figure 8 As described.

[0085] from Figure 3 Starting at frame B102, method 100 includes receiving and transmitting camera data (CC) as a camera feed set via camera 16. 16 Camera data (CC) 16 This includes the QR code 24 from camera 16 as vehicle 12 travels toward infrastructure node 11. Figure 1 The digital pixel image of the camera (CC). Then, the camera data (CC). 16 The input is provided to box B104 for visual processing.

[0086] Box B103 requires the use of optional crowdsourcing to select possible QR code locations as used along... Figure 1 Candidates for optimal placement of road surface 13 in area 20. Brief reference. Figure 1 The camera control system 25 may seek to locate specific areas or zones for use as the optimal placement area 20. Therefore, an embodiment is conceivable in which the camera control system 25 uses historical data from previous drive-through events to determine candidate areas for this purpose. Box B103 may include providing such historical data as input to method 100, in this case to box B106.

[0087] Box B104 includes camera data (CC) from box B102 processed via a computer vision module or by using a suitable object detection algorithm. 16 For example, box B104 might need to use a Region Proposal Network (RPN) to generate proposals for regions of interest (CC) for extracting camera data. 16 Potential candidate regions in ) . Method 100 continues after transferring the results of box B104 to box B106.

[0088] Box B105 needs to store the informed QR code locations of previously scanned QR codes 24, such as locations that have been used a certain number of times within a specific duration. These locations are then provided as input to box B106.

[0089] At box B106, Figure 3 Method 100 can process the outputs of boxes B103, B104, and B105 to generate a set of candidate QR code regions for further consideration. For example, camera control system 25 can determine... Figure 1 The possible locations of the optimal placement area 20. This set of candidate QR code areas is provided to box B108 as the output of box B106.

[0090] At box B108, method 100 next includes adjusting one or more camera settings for optimal QR code scanning of the event in real time, doing so for each region in the candidate QR code region as the vehicle 12 approaches and passes through it. Generally, if the camera data (CC...) 16 If the calibration standard is unclear, the camera parameters are tuned at box B108. Then, method 100 proceeds to box B109 to continue the QR code health check according to this disclosure. As described below, box B108 also receives input from box B112. A representative method 400 for adjusting camera settings is referenced below. Figure 9 describe.

[0091] Box B109 includes performing a multifactor QR code health check. As mentioned above, box B109 may require evaluation of two criteria against a calibration threshold: (1) camera data (CC). 16 The clarity of (2) the “skewness” or angular orientation of the center line (LL) of the QR code module 30 (see Figure 6 If the camera control system 25 determines that both criteria have been met, method 100 proceeds to block B111. If the camera control system 25 determines that one or both criteria have not been met, method 100 alternatively proceeds to block B112. An embodiment of multifactor health examination is described as method 200 in... Figure 4 The diagram is shown below, and further detailed description is provided in the following text.

[0092] Continue to refer to Figure 3Method 100, box B110 includes collecting crowdsourcing details as a form of guided QR code discovery. For example, when accessing infrastructure node 11, the current settings of the corresponding camera 16 of vehicle 12 can be shared for each successful scan of QR code 24 by another vehicle 12 (i.e., another vehicle 12 in a fleet of vehicles 12). As an example and not a limitation, successful history scan details in a crowdsourcing scenario may include the time of the scan, store name / location, QR code details, and vehicle details. QR code details may include, for example, the three-dimensional (3D) position and orientation of the displayed QR code 24, the skewness of the QR code 24 at the successful scan location, and the pixel count per module at the successful scan location.

[0093] Vehicle details may further include, for example, camera parameters, the 3D pose of vehicle 12 / camera 16, the specific brand / model of camera 16, and the GPS location of vehicle 12 at a successfully scanned location. In general, crowdsourced information acts as a mechanism for the camera control system 25 regarding... Figure 1 Is the QR code 24 too far, too high, too low, or otherwise not optimally positioned for guidance in a successful scan from the camera 16? Once the crowdsourcing details are received, method 100 proceeds to box B112.

[0094] After a successful multi-factor health check, the vehicle proceeds from box B109 to box B111. In this case, the camera control system 25 can continue decoding the QR code 24 and complete the QR code-assisted vehicle-merchant transaction.

[0095] exist Figure 3 At box B112, the camera control system 25 responds to unsuccessful multifactor health checks at box B109 with selective skewness and distance control as required. For example, if the vehicle 12 is too far from the displayed QR code 24 and / or the approach angle relative to the displayed QR code 24 is too significant to complete the scan successfully, the camera control system 25 can, for example, adjust the skewness and distance control accordingly. Figure 8 The representative method 300 performs skew and distance control.

[0096] At box B114, the camera control system 25 can determine that the QR code scanning process is destined to fail, regardless of the actions taken in box B112. In other words, the camera control system 25 can assess the severity of the skew or the distance to the QR code 24 and consider extreme skew / distance as corresponding to a "destined failure" condition. When this occurs, the camera control system 25 can retrieve the information from its memory 27. Figure 2 Set the bit code in ) and continue to box B116.

[0097] At box B116, the camera control system 25 can prompt the operator of vehicle 12 to perform a user action. For example, the camera control system 25 can present a message via an infotainment system screen (not shown) located inside vehicle 12, possibly with instructions to scan QR code 24 using a smartphone camera or another QR code scanning device. Method 100 ends when the scan is successfully performed using an alternative means.

[0098] QR code multi-factor health check: Refer now Figure 4 The diagram illustrates what can be used to execute Figure 3 Method 200 for box B109. Begins in box B202 (similar to...). Figure 3 (B108) When vehicle 12 approaches QR code 24, camera control system 25 can begin to adjust the parameters of camera 16.

[0099] Then, method 200 continues to box B204, where the camera control system 25 performs a functional pattern recognition process. As will be appreciated in the art, similar to... Figure 1 The QR code 24 has a variety of patterns that collectively implement the encoding function. Such patterns include (1) a finder or positioning pattern to help locate the QR code, (2) an alignment pattern to help the camera 16 successfully read the QR code 24 from an oblique angle, (3) a timing pattern to help divide the data modules of the QR code into a grid system, and a quiet zone bordering the remainder of the QR code 24, which helps to isolate the QR code 24 from interference from its surrounding environment. When no functional pattern is identified at box 204, method 200 proceeds to box B205, and when a functional pattern is successfully identified, method 200 alternatively proceeds to box B206.

[0100] Box B205 includes waiting for vehicle 12 to move closer to QR code 24, and returning to box B202, thereafter repeating box B202 as described above.

[0101] Box B206 includes a health check (i.e., skewness check) of the centerline of the startup module, wherein, as understood in the art, each module is a sub-region of the complete QR code 24.

[0102] Brief Reference Figure 5 ,when Figure 1 Vehicle 12 approached Figure 1 When the infrastructure node 11 is in use, the QR code 24 can be arranged relative to the camera 16 at a viewing angle (α). This arrangement allows the square shape of the QR code 24 to appear as a non-standard shape, such as a rhombus or a long rectangle.

[0103] Figure 6The illustration shows the effect of skewness on the center line LL of each module 30 of the QR code 24, which will manifest as a deviation from the true center of a given module 30. When directly viewing the QR code 24, each module 30 typically has a square shape (i.e., no skewness), due to... Figure 5 From the perspective (α), each module 30 appears as an elongated rectangle. Within each module 30 is an aperture 31, which, in a normal / orthographic view (i.e., a non-skewed view), would be located at the exact center of the (square) module 30. However, conversely... Figure 6 The aperture 31 is offset by a corresponding distance (i.e., length l) from the corresponding left edge 32L and right edge 32R. l and l r This can cause inaccurate QR code scanning results. A health check can be used for QR codes, such as by comparing two colors to determine if they are the same: (i) the color of aperture 31, and (ii) the color of the surrounding module 30 of aperture 31. If both colors fall at the same end of the histogram, module 30 is considered "healthy." If the colors fall at opposite ends of the histogram spectrum, module 30 is considered "unhealthy." Once the health check has begun, method 200 proceeds to box 207.

[0104] At box B207, the camera control system 25 can compare the offset with a threshold (θ) to determine whether the skewness requires vehicle placement correction. Figure 4 In the middle, the center line LL ( Figure 6 The absolute values ​​of the left and right offsets are compared with a threshold (i.e., |l l -l r |>θ). When the threshold (θ) is exceeded, method 200 continues to box B208, while when the threshold (θ) is not exceeded, method 200 instead continues to box B212.

[0105] At box B208, the camera control system 25 can prompt the driver of vehicle 12 to move vehicle 12 to a new position to reduce the offset from box B207. In other embodiments, box B208 can be executed autonomously, for example, when vehicle 12 is operated autonomously. As a possible feature, the camera control system 25 can track the offset and automatically decode the QR code 24 when the offset is less than a threshold (θ).

[0106] Figure 4 Box B210 includes the option to perform a distance health check. (Brief Reference) Figure 7A , Figure 7B and Figure 7C QR code 24 at a given moment can be located away from camera 16 ( Figure 7A At a distance of 16 from the camera ( Figure 7B ), or get close to the camera 16 ( Figure 7C The farther the QR code 24 is from the camera 16, the smaller it appears in the camera 16's reference frame, which translates to fewer pixels being used. Assuming no view skew, the distance between the camera 16 and the QR code 24 may still require parameter tuning to obtain optimal scanning performance.

[0107] exist Figure 7A , Figure 7B and Figure 7C In each of the corresponding distance cases, the corresponding pixel bars 30A, 30B, and 30C represent the width of a single pixel. Therefore, in a given image, the same object positioned at a greater distance requires fewer pixels. However, for Figure 7A In the farthest example, QR code 24 is too small relative to a pixel width of 30A, making accurate scanning of QR code 24 difficult. Figure 7B and Figure 7C The scannable area of ​​QR code 24 is gradually increased, thus improving the scanning results.

[0108] Therefore, in box B210, the camera control system 25 can monitor the pixel count to see if there are enough pixels visible to the camera 16. If enough pixels are visible, method 200 can proceed to box B212; otherwise, method 200 returns to box B205.

[0109] At box B212, the camera control system 25 can register a bit code indicating whether the skewness and pixel count are good / sufficient for accurate scanning of the QR code 24. Then, method 200 proceeds to box B214, where the camera control system 25 decodes the QR code 24 and continues with the vehicle-merchant transaction.

[0110] Distance processing: can be based on Figure 3 Exemplary method 300 within the scope of method 100 is used to process vehicle 12 relative to Figure 1 The placement of the display screen 22 / infrastructure node 11. The camera control system 25 can determine how many modules 30 are in the version (v) QR code 24. Figure 6 As understood in the art, QR code versions range from 1 to 40, and describe the size and data capacity of QR code 24. For example, version 1 is the smallest, with 21 modules x 21 modules, while version 40 is the largest (177 x 177). Therefore, the QR code version (v) describes the total grid size, maximum data storage capacity, and other information. Thus, the module count (C) is version-specific, i.e., C = 4v + 17.

[0111] refer to Figure 8Method 300, box 302 may require the use of camera control system 25 to perform the functional pattern recognition process. This box is functionally similar to that described above. Figure 4 Box B204. Once the functional pattern is recognized, method 300 proceeds to boxes B304 and B305.

[0112] Box B304 includes removing pixels associated with the function pattern in box B302. Box B304 may require, for example, creating a digital mask near the function pattern pixel location and zeroing / ignoring the masked pixels during data decoding. In this way, data module 30 is better isolated. Method 300 then proceeds to box B306.

[0113] Box B305 includes determining the QR code version (v), which, as will be appreciated in the art, is presented as part of the QR code 24 itself. Method 300 then proceeds to box B308.

[0114] At frame B306, the camera control system 25 records a value (P) that describes the total number of remaining pixels in the data module 30 after the functional pattern is removed at frame B304. Thereafter, method 300 continues to frame B308.

[0115] continue Figure 8 The discussion states that frame B308 needs to calculate the pixel count per module based on the QR code version (v) (i.e. ). Implementations of box B308 may include using processor 26 ( Figure 2 Calculate the value using the following formula:

[0116]

[0117] Here, again, v is the QR code version, P is the number of remaining pixels in data module 30 after the functional pattern is removed at box B304, and C is the module count (i.e., 4v+17 as described above). Method 300 then proceeds to box B310.

[0118] exist Figure 8 At frame B310, the camera control system 25 determines the pixel count per module (i.e. Is the pixel count sufficiently high? Box B310 may include comparing the calculated value of the pixel count per module with a predetermined threshold. When the pixel count per module is less than the threshold, method 300 proceeds to box 312, and when the pixel count per module is equal to or exceeds the threshold, method 300 alternatively proceeds to box B314.

[0119] Box B312 includes determining, via camera control system 25, that vehicle 12 should move closer to QR code 24. In one or more embodiments, the decision at box B312 may be made by crowdsourced data 35 (e.g., from...). Figure 3 The notification from box B110. The control response generated from box B312 may include waiting for an appropriate duration before repeating method 300, and may include a reminder. Figure 1 The driver's QR code 24 of vehicle 12 is not close enough. An autonomous embodiment of vehicle 12 can automatically control camera 16 and vehicle 12 to achieve the same effect. Once vehicle 12 has moved to a better position for scanning, method 300 is complete.

[0120] Box B314 includes decoding the QR code 24 via processor 26 when the pixel count per module exceeds a predetermined pixel count per module threshold. As appreciated in the art, decoding involves... Figure 2 The processor 26, or more likely a dedicated image processor of the camera 16, performs various image processing techniques. These techniques may include, for example, noise removal, edge detection, pixel value extraction, error correction, etc., then converting binary data into a machine-readable format and interpreting the encoded information. Figure 1 In typical scenarios, decoding can lead to the initiation / completion of vehicle-merchant transactions, such as ordering goods or services, paying bridge tolls or road fees, etc. Once QR code 24 is decoded, method 300 is complete.

[0121] Tuning camera parameters: Reference Figure 9 The method 400 for adjusting or tuning camera parameters can be used in one or more embodiments to achieve... Figure 3 The method 100 shown in the figure is box B108. Generally, if Figure 2 Camera data (CC) 16 If the parameters are unclear, one or more parameters can be adjusted in real time.

[0122] Starting from box B401, once QR code 24 has been... Figure 1 Once camera 16 scans, camera control system 25 can execute one or more of the following boxes B401 (sharpness control), B403 (brightness control), B405 (size), and B406 (viewing angle) to initiate tuning of one or more parameters. As described above, and as performed at box B407, in one or more embodiments, crowdsourced QR code location information can be used to adjust some of the parameters (e.g., size and viewing angle). Boxes B401, B403, B405, and B406 continue to the corresponding boxes B402, B404, B406, and B408.

[0123] The path extends from box B401 (sharpness control) to box B402, and therefore box B402 may need to be configured. Figure 1 and Figure 2 The shutter speed of camera 16. The shutter speed depends on ambient lighting conditions and other factors, such as the lens length used by camera 16. By increasing or decreasing the shutter speed as needed, camera control system 25 can change the exposure time and minimize image blur. Thereafter, method 400 continues to box B412.

[0124] The path extends from frame B403 (brightness control) to frame B404, and therefore frame B404 may need to be configured. Figure 1 and Figure 2 The camera 16's shutter speed and / or aperture size. Various adjustments can be used to increase or decrease image brightness, for example, by changing the aperture size, gamma correction, histogram equalization, shutter speed, etc. Thereafter, method 400 continues to box B412.

[0125] From box B405 (size), move to box B406, and therefore box B406 may need to be adjusted. Figure 1 and Figure 2 The digital and / or optical zoom level of camera 16. Alternatively or simultaneously, frame B406 may include controlling the movement of vehicle 12 or prompting the driver of vehicle 12 to continue driving for a predetermined duration or distance before resampling QR code 24. Thereafter, method 400 continues to frame B412.

[0126] From frame B406 (viewpoint) to frame B408, and therefore frame B408 may need to control the movement of vehicle 12 or prompt the driver of vehicle 12 to continue driving for a predetermined duration or distance before resampling QR code 24. Alternatively or simultaneously, frame B408 may include controlling the panning settings of camera 16 to change the viewpoint, and then, when camera 16 is so configured, transmitting camera control signals to camera motor 160 of camera 16. Thereafter, method 400 continues to frame B412.

[0127] exist Figure 9 At box B412, the camera control system 25 can resample or rescan the QR code 24, and then, based on the result of the rescan, extract from box B108 ( Figure 3 )continue.

[0128] Method 100 described above, along with its supporting methods 200, 300, and 400, can be used in... Figure 1The system is used on vehicle 12 to achieve accurate QR code scanning using the vehicle's camera suite (e.g., one or more cameras 16). Various teachings selectively implement real-time tuning of camera parameters and health checks of QR codes generated by the infrastructure during drive-through / drive-by vehicle-to-merchant transactions. Such adjustments can be coupled with optional autonomous driving control actions or HMI-based driver cues to correct the position of vehicle 12 relative to infrastructure node 11. Furthermore, crowdsourcing is used to optimize the solution so that camera control system 25 can identify the correct scanning location for accurate scanning and decoding of QR codes when displayed by infrastructure node 11. Those skilled in the art who have benefited from the foregoing disclosure will now appreciate these and other accompanying benefits of this teaching.

[0129] This disclosure allows for many different forms of embodiments. Representative examples of this disclosure are shown in the accompanying drawings and are described in detail herein as non-limiting examples of the disclosed principles. Therefore, elements and limitations described in the abstract, background, summary, and detailed description sections, but not expressly set forth in the claims, should not be incorporated into the claims, individually or collectively, by implication, inference, or otherwise.

[0130] For the purposes of this specification, unless otherwise stated, the use of the singular includes the plural, and vice versa; the terms “and” and “or” should be both conjunctions and disjunctive words; “any” and “all” should both mean “any and all”; and the words “including,” “containing,” “comprising,” “having,” etc., should mean “including but not limited to.” Furthermore, approximate words such as “approximately,” “almost,” “substantially,” “generally,” “approximately,” etc., may be used herein in the sense of “being, near, or almost being,” or “within 0-5%,” or “within acceptable manufacturing tolerances,” or logical combinations thereof.

[0131] The detailed description and accompanying drawings are supporting and descriptive of this teaching, but the scope of this teaching is defined only by the claims. While the best mode and some other embodiments for carrying out this teaching have been described in detail, various alternative designs and embodiments exist for practicing the teaching as defined in the appended claims. Furthermore, this disclosure explicitly includes combinations and sub-combinations of the elements and features presented above and below.

Claims

1. A method for scanning a Quick Response (QR) code during a vehicle-to-merchant transaction, comprising: When the QR code is displayed on the display screen of the infrastructure node, as the vehicle moves relative to the display screen, camera data of the QR code is collected using a camera mounted on the vehicle; The camera data is transmitted to the processor of the camera control system; The camera data is processed by the processor of the camera control system to determine the characteristics of the camera data; Perform the multi-factor health check using the QR code; as well as In response to one or more of the characteristics having a value less than the corresponding threshold of the multifactor health check, control actions are performed on the camera and / or the vehicle.

2. The method of claim 1, wherein the factors of the multi-factor health check include the clarity of the image data, and wherein the control action includes adjusting the camera settings to optimize the scanning of the QR code.

3. The method of claim 2, wherein the factors of the multi-factor health check include the viewing angle of the QR code, and wherein the control action includes adjusting the panning setting and / or tilt setting of the camera when the viewing angle exceeds a predetermined viewing angle as the corresponding threshold.

4. The method of claim 3, wherein performing the control action includes causing a change in the position of the vehicle relative to the QR code when the viewing angle exceeds a viewing angle threshold.

5. The method of claim 4, wherein the vehicle is an autonomous vehicle, and wherein the change in the position of the vehicle relative to the QR code includes controlling the steering and / or propulsion state of the autonomous vehicle via the processor.

6. The method of claim 1, further comprising: Receive crowdsourced data indicating the previous scan location of the QR code; The crowdsourced data is processed by the processor of the camera control system to determine the optimal placement area for the vehicle; as well as Control the state of the vehicle so that it is guided to the optimal placement area.

7. The method of claim 1, further comprising: The processor determines the QR code version. Calculate the pixel count for each module based on the QR code version; as well as When the pixel count per module exceeds a predetermined pixel count threshold per module, the QR code is decoded by the processor.

8. A vehicle comprising: Vehicle body; The wheel assembly connected to the vehicle body; as well as A camera control system operable to scan Quick Response (QR) codes during vehicle-to-merchant transactions between the vehicle and infrastructure nodes, the camera control system comprising: Car-mounted camera; Processor; and A computer storage medium ("memory") having instructions stored thereon, which can be executed by the processor to cause the camera control system to perform the following operations: When the QR code is displayed on the display screen of the infrastructure node and the vehicle moves relative to the display screen, camera data of the QR code is collected via the vehicle-mounted camera; The camera data is processed by the processor to determine the characteristics of the camera data; Perform a multi-factor health check on the QR code, including (i) checking the clarity of the image data and (ii) checking the viewing angle of the QR code; In response to one or more of the characteristics being less than the corresponding threshold of the multifactor health check, the camera control action is performed by adjusting one or more settings of the camera to optimize the scanning of the QR code.

9. The vehicle of claim 8, wherein the vehicle-mounted camera includes a camera motor operable to control the panning level and / or tilt level of the vehicle-mounted camera, and wherein the instructions are executable by the processor to cause the camera control system to adjust the panning setting and / or tilt setting of the camera via the camera motor when the viewing angle exceeds a predetermined viewing angle.

10. The vehicle of claim 8, wherein the vehicle is configured as an autonomous vehicle, and wherein the instructions are executable by the processor to cause an autonomously controlled change in the position of the vehicle relative to the QR code when the viewing angle exceeds a viewing angle threshold.

Images