Method and apparatus for determining visibility

Abstract

Embodiments of the present disclosure relate to methods and apparatuses for determining visibility. Described herein is a method of determining visibility of predefined content in a display displaying dynamic background content. The method can comprise, after determining that the predefined content is to be overlaid on the dynamic background content for presentation in the display, obtaining at least one image representing visual information displayed on the display; and performing computer vision based processing based on the obtained at least one image to determine the visibility of the predefined content in the display.

Description

Cross-references to related applications

[0001] The disclosure of German patent application No. 102024136072.0, filed on December 4, 2024, including the specification, drawings and abstract, is incorporated herein by reference in its entirety. Technical Field

[0002] This invention relates generally to techniques related to content visibility inspection, and more specifically to techniques related to determining the visibility of predefined content on a display with a dynamic background. Background Technology

[0003] In recent years, cabin user experience (UX) has become one of the key differentiators among automotive brands. Wide, panoramic, transparent, or similar displays are impressive features for both automakers and customers. For example, augmented reality (AR) technologies, which typically involve overlaying real-world images of the environment with information on the driver's navigation or infotainment displays, are a growing trend in the automotive industry.

[0004] However, safety is often relegated to a secondary position when presenting visual information, although it is certainly still crucial for the driver. More specifically, displaying warning signs (such as tell-tales / Advanced Driver Assistance Systems (ADAS) indicators) on transparent screens with dynamic background images is often an underestimated challenge. On such (e.g., transparent) displays, it is sometimes necessary to show warning signs. To ensure driver safety, it must be ensured that the warning signs are actually displayed at the end of the image presentation process and are visible to the driver.

[0005] While cyclic redundancy check (CRC) based schemes are commonly used in classic instrument clusters to verify the correct display of warning lights, this method is unsuitable for dynamic background displays because they will frequently trigger false alarms. On static displays, this is typically accomplished by obtaining the CRC code of the warning image and confirming its correctness. However, this is usually impossible on dynamic displays because the background changes dynamically, affecting the CRC code.

[0006] Therefore, there is a need for improved mechanisms, such as methods, apparatus and / or systems, that can reliably and efficiently verify the correct appearance or visibility of predefined content (e.g., indicators, warning signs, alert lights, etc.) on displays with dynamic backgrounds (e.g., transparent ones). Summary of the Invention

[0007] In view of some or all of the above-mentioned technical needs, this disclosure generally provides a method, corresponding apparatus, program, and computer-readable storage medium for determining the visibility of predefined content in a display showing dynamic background content, having the features of the respective independent claims.

[0008] According to one aspect of this disclosure, a method is provided for determining the visibility (or occurrence) of predefined (or pre-determined) content in a display (e.g., a screen, monitor, etc.) displaying dynamic background content. It is noteworthy that throughout this disclosure, such predefined / pre-determined content can be in any suitable form, such as, but not limited to, warning signs, indicators, symbols, warning lights, text, etc. This is not limited to this disclosure. Furthermore, the display can be implemented in any suitable form, such as, but not limited to, the display (screen) of a handheld device, a computer, a vehicle (e.g., an automobile), etc.

[0009] Therefore, dynamic background content can include any suitable (background) information displayed (presented) on the display. As an illustrative (non-limiting) example, in the case of an automotive display, the background content / information could involve, for example, navigation information or even infotainment information such as music, video, etc.

[0010] Specifically, the method may include: after determining that predefined content will be overlaid on dynamic background content for rendering on a display, obtaining at least one image representing visual information displayed on the display. The method may also include: performing computer vision-based processing based on the obtained at least one image to determine the visibility of the predefined content on the display (or in other words, whether the predefined content is visible on the display). As will become apparent from the following more detailed description, depending on the various implementations and / or contexts, any suitable computer vision-related techniques may be employed here, individually or in combination.

[0011] By this designation, this disclosure generally aims to provide an efficient, flexible, and reliable mechanism capable of checking / determining the appropriate presence or display of specific (predefined) content (e.g., warning signs) in a display system with a dynamic background (e.g., in a vehicle) (even where the display may be transparent). It is noteworthy that in some possible situations, such as in the automotive industry, ensuring that such content (e.g., warning signs / indicators, warning lights, etc.) is correctly displayed and visible to the driver can be considered critical for safety, regulatory compliance, and user experience. As previously stated, conventional methods (e.g., CRC-based techniques) may be considered insufficient in certain use cases or scenarios (such as displays with dynamic backgrounds where content is mixed into the dynamic background with a certain degree of transparency), leading to potential safety risks and regulatory non-compliance.

[0012] In some example embodiments, the at least one image may be obtained from a video stream as one or more image frames representing (visual) information to be presented on a display. As those skilled in the art will understand and recognize, the at least one image may be obtained in any other suitable manner, such as by taking a screenshot / snapshot of the display.

[0013] In some example embodiments, a video stream can be generated by mixing (or overlaying) frames representing predefined content (e.g., image or video frames) with frames representing dynamic background content (e.g., image or video frames).

[0014] In some example embodiments, the output (result) of visibility determination may include information indicating whether the predefined content is visible on the display or not.

[0015] In some example embodiments, if it is determined that predefined content is not visible on the display, the method may further include comparing the color of the predefined content with the background color of the area (of the display) where the predefined content will (or should) be presented / displayed. In some other possible examples, if it is determined that the background color in the region of interest is relatively similar to the color of the predefined content (e.g., a red warning sign at least partially overlaps with a red car shown in the background, resulting in a false alarm for a computer vision algorithm), in some possible implementations it may be possible to consider changing the color of the predefined content (e.g., changing it from red to green) for reprocessing to re-examine the presence of the predefined content on the display. Of course, as those skilled in the art will understand, any other suitable processing may also be applied here. For example, at least in theory, it may also be possible to consider (temporarily) changing the background color in the region (region of interest) in a suitable manner and repeating the entire process used to determine the visibility of the predefined content.

[0016] In some example embodiments, if it is determined that predefined content is not visible on the display, the method may further include generating a signal (e.g., a warning signal or the like) indicating that the predefined content is not visible on the display. This signal may be notified (e.g., visually or audibly) to at least one of the following: relevant system components / entities (software or hardware), interested persons (e.g., end users, vehicle drivers, etc.). Of course, as those skilled in the art will understand and recognize, depending on the various implementations and / or contexts, any other suitable (safety) measures may be employed.

[0017] In some example embodiments, computer vision-based processing may be at least partially based on artificial intelligence (AI). As those skilled in the art will understand and appreciate, depending on the various implementations and / or contexts, any suitable AI-based technology may be employed. For example, techniques involving machine learning, neural networks (NNs), etc., may be implemented. Of course, even when used in conjunction with suitable AI-based technologies, any other suitable traditional, classical approach may be employed.

[0018] In some example embodiments, performing computer vision-based processing may include generating shape features within a region of interest (ROI) on a display, where predefined content will (or should) be rendered / displayed. In some possible examples, shape features may include, but are certainly not limited to, information indicating at least one of the following: contour, area, or connectivity. As will be understood and appreciated by those skilled in the art, the generation of shape features may involve techniques related to edge generation, such as cany, sobel, hough transform, wavelets, etc., depending on various implementations and / or contexts.

[0019] In some example embodiments, prior to generating shape features, the method may further include receiving an indication of a region of interest within the display, in which predefined content will be presented, displayed, or shown. Depending on the implementation, such a region of interest (or more generally, the area within the display in which predefined content should be presented / displayed) may be known a priori, such as predefined, preconfigured, or even prescribed (e.g., by standards). In some possible examples, regardless of the availability of such an indication of the region of interest, computer vision-based processing may involve scanning the entire area of ​​the display and searching for such predefined content for generating shape features of said predefined content.

[0020] In some example embodiments, the computer vision-based processing may further include preprocessing the region of interest before generating these shape features. Such preprocessing may involve at least one of the following: color conversion operation, binarization operation, or morphological operation. Of course, as those skilled in the art will understand and recognize, any other suitable preprocessing technique may be employed depending on the various implementations and / or contexts.

[0021] In some example embodiments, performing computer vision-based processing may further include comparing the generated shape features with shape features (base facts) of (expected) predefined content to generate one or more metrics indicating the difference between the generated shape features and the shape features (base facts) of the (expected) predefined content (i.e., between the previously generated shape features and those expected safe contents being inspected). Again, this can be achieved using any suitable method / algorithm, such as template matching, cross-correlation, etc.

[0022] In some example embodiments, performing computer vision-based processing based on at least one obtained image to determine the visibility of predefined content on the display may further include determining the visibility of the predefined content on the display based on one or more generated metrics. Specifically, this determination may involve techniques based on machine learning (e.g., principal component analysis, support vector machines, etc.) and / or decision matrices. For example, in some possible examples, machine learning techniques may be applied to previously generated metrics to classify these metrics and make decisions about whether or not to present the predefined content within a region of interest on the display (e.g., using a decision matrix).

[0023] In some example embodiments, computer vision-based processing may be performed repeatedly (iteratively) on multiple images. Therefore, determining the visibility of predefined content on a display may also involve majority voting on the corresponding outputs of the computer vision-based processing performed on multiple images. For example, in some possible examples, the computer vision-based processing may be repeated multiple times on the predefined content to perform majority voting in a classifier decision, thereby increasing the confidence level of the final decision on the visibility of the predefined content.

[0024] In some example embodiments, predefined content can be rendered on a display using a graphics processing unit (GPU), such as the GPU of a handheld device, vehicle, etc. Of course, any other suitable (hardware) component / module can also be used, depending on the various implementations and / or contexts.

[0025] In some example embodiments, predefined content may include safety-related content such as warning signs, hazard lights, etc. Of course, depending on the implementation and / or context, any other previously indicated, suitable safety-related content is also possible. For example, in some possible examples, other suitable graphic content (such as a speedometer pointer) may also be considered safety-critical, or gear indicators P, D, N, etc., speed markings, etc. This is not limited to this disclosure.

[0026] In some example implementations, the predefined content includes text content (e.g., text strings, text blocks, etc.). Of course, in some possible cases, this text content can also be security-related (e.g., security-related or warning text).

[0027] In some example embodiments, the display may be a vehicle display. Of course, as mentioned above, any other suitable display setup may be used depending on various implementations and / or requirements.

[0028] In some example embodiments, the display may be a display with a dynamic background, in which content is transparently blended into the dynamic background. Such a display (or sometimes simply referred to as a "transparent" display, etc.) may be of particular interest to the techniques proposed in this disclosure, one reason being that conventional (e.g., CRC-based) techniques tend to cause frequent false alarms (e.g., due to small deviations in pixels) on such (transparent) displays that exhibit dynamic (e.g., frequently changing) background information / content.

[0029] In some example embodiments, displaying dynamic background content may involve overlaying augmented reality (AR) images of the environment. For example, such AR-related techniques may involve overlaying (blending) a real-time image of the environment with information on a driver's navigation or infotainment display, etc. Of course, as those skilled in the art will understand and recognize, any other suitable techniques for displaying dynamic backgrounds on a display can be implemented.

[0030] According to another aspect of this disclosure, an apparatus is also provided. In particular, the apparatus may include suitable components, such as a processor and memory coupled to the processor. The processor may be adapted to cause the apparatus to perform any of the example methods described throughout this disclosure.

[0031] According to another aspect of this disclosure, a (computer) program is provided. This computer program may include instructions that, when executed by a processor, cause the processor to perform any of the example methods described throughout this disclosure.

[0032] According to another aspect of this disclosure, a computer-readable storage medium is provided. This computer-readable storage medium can store the aforementioned computer program.

[0033] As those skilled in the art will understand, the details of the disclosed methods can be implemented as a system (e.g., in the form of an apparatus) suitable for performing some or all of the steps of the method, and vice versa. In particular, it should be understood that the methods according to this disclosure relate to methods of operating a system (or apparatus) according to the above embodiments and variations thereof, and the corresponding statements made with respect to the system (or apparatus) also apply to the corresponding methods, and vice versa.

[0034] It should also be understood that, in this disclosure, the terms “coupled” or “coupled” refer to elements that are electrically communicating with each other, whether directly connected, for example, via a wire or in some other way (e.g., indirectly). Notably, an example of coupling is a connection. Attached Figure Description

[0035] The following explanation of exemplary embodiments of this disclosure is based on the accompanying drawings, wherein the same reference numerals denote the same or similar elements, and in the drawings:

[0036] Figure 1 Examples of possible implementations of the proposed technology according to some possible exemplary embodiments of this disclosure are illustrated schematically.

[0037] Figure 2 The illustrations illustrate simplified implementations or working principles of the proposed techniques according to some possible exemplary embodiments of this disclosure;

[0038] Figure 3 The illustrations illustratively depict examples of possible implementations of computer vision-based processing according to some possible exemplary embodiments of the present disclosure, and

[0039] Figure 4 This is a flowchart illustrating, schematically, an example of a method for determining the visibility of predefined content in a display showing dynamic background content, according to some possible exemplary embodiments of this disclosure. Detailed Implementation

[0040] As stated above, unless otherwise specified, the same or similar reference numerals in this disclosure may indicate the same or similar elements, such that repeated descriptions thereof may be omitted for the sake of brevity. Furthermore, it should be noted that, unless otherwise stated, the symbols used in the drawings are for illustrative purposes only and should not be construed as constituting any kind of limitation.

[0041] As mentioned above, in recent years, cabin user experience (UX) has become one of the key differentiators among automotive brands. In particular, wide, panoramic, and transparent displays are attracting both automakers and consumers / customers. For example, augmented reality (AR) technology (which involves overlaying a realistic image of the environment onto information on the driver's navigation or infotainment display) is a growing trend in automobiles. Warning signs need to be displayed on such displays. To ensure driver safety, these warning signs must be displayed correctly and be visible to the driver.

[0042] However, while displaying visual information remains crucial for drivers, safety is often relegated to a secondary position. More specifically, among other things, the challenge of displaying warning signs, hazard lights, and / or Advanced Driver Assistance Systems (ADAS) related content / information on such screens with dynamic background images (e.g., transparent ones) is often underestimated.

[0043] In classic instrument clusters, cyclic redundancy check (CRC) schemes are typically used to identify the proper display of symbols or indicators (e.g., warning lights). However, such methods cannot be readily applied to dynamic background displays because they can frequently generate false alarms (e.g., due to small deviations in pixels). More specifically, on typical static displays, verification can usually be accomplished by obtaining the CRC code of the warning sign image and confirming its correct visibility. However, this method is impractical on dynamic displays because the background is constantly changing.

[0044] Therefore, in general, this disclosure proposes a mechanism for checking the proper display of such dynamic content (e.g., warning signs, warning lights, etc.) even on transparent displays. More broadly, this disclosure generally describes a method for feeding back (or relaying) images to a computer vision-based (sub)system that analyzes, for example, image recognition, whether content (e.g., warning signs, warning lights, etc.) is properly displayed and visible. Among other things, this method can be considered beneficial in ensuring the functional safety (FuSa) compliance of such dynamic displays.

[0045] However, it is worth noting that, as those skilled in the art will understand and recognize, although this disclosure may appear to consistently relate to warning signs, hazard lights, etc., these are merely some possible examples for illustrative purposes and should therefore not be construed as constituting any kind of limitation. Also, as previously indicated, the techniques proposed in this invention can be adapted to check / determine the visibility (or presence) of any other suitable (predetermined or predefined) content within a display, which may include, but is not limited to: indicators, text information (e.g., text strings, text blocks, etc.). Similarly, although an automotive display can be used as an example for illustration, any other suitable display (transparent or opaque) such as a handheld device, another type of vehicle, etc., can also be used, depending on the various implementations and / or contexts.

[0046] Now refer to the attached diagram, Figure 1 Examples of possible implementations of a system 100 employing the proposed technology according to some possible exemplary embodiments of this disclosure are illustrated schematically. Figure 1 In this example, system 100 is schematically shown for verifying the proper display of dynamic warning lights on a transparent AR-authorized vehicle display. Generally, this example system 100 can be understood as utilizing computer vision technology to ensure that (predefined) critical warning indicators (such as "ESP off" in this example) are accurately displayed and visible against a dynamic background, thereby meeting functional safety requirements and enhancing driver safety. As mentioned above, in Figure 1In this document, the vehicle (car) display and the warning sign “ESP off” are used only as possible examples for illustrative purposes and should therefore not be considered as constituting any kind of limitation.

[0047] Specifically, such as Figure 1 As illustrated in the illustration, this process is typically handled by module 101 in the vehicle (e.g., such as...). Figure 1 The process begins when the real-time core (or processor) module, or any other suitable software / hardware-based module / entity, shown in the example receives a request to display (or present) a specific (predefined) warning indicator, such as "ESP Off" (Electronic Stability Program Off) in the current situation. Therefore, the "ESP Off" warning light can be understood as indicating that the predefined content for which protection is sought needs to be correctly verified for visibility within the display. Depending on various implementations and / or scenarios, this request can be generated and received by any suitable external module of the vehicle (not shown). This request then triggers the system to begin the entire verification process to determine whether the requested dynamic warning sign is correctly displayed and visible on the display (shown with background information based on dynamic AR). It should be noted that in Figure 1 In this example, the real-time core / processor module 101 may also appear to be responsible for ultimately concluding the process by verifying visibility based on results from the computer vision-based module 107. Of course, as those skilled in the art will understand and recognize, this final verification / determination step can be well performed by another suitable (software / hardware based) module / entity different from the real-time core / processor module.

[0048] Upon receiving an initial request from the real-time core / processor module 101, the GPU 102 can then be configured to initiate the rendering of enhanced content. Figure 1 In this example, the GPU includes two main components: a Safety Rendering (ASIL - B) component, which can be understood as responsible for rendering safety-critical elements (in this example, the "ESP Off" warning) with high integrity, thereby ensuring that the safety-critical elements meet automotive safety standards; and a 3D / 2D Rendering (QM) component, which can be understood as managing general rendering tasks to create visual content, such as (enhanced) navigation information and possibly other display elements. Therefore, the output of GPU module 102 can be considered as including both a dynamic background image and warning icons, which are appropriately blended (overlaid) for presentation to be ultimately displayed to the end user of the vehicle (e.g., the driver or passengers).

[0049] The presented image can be temporarily stored in dynamic random access memory (DRAM) 103 or any other suitable storage module that may contain multiple frame buffers for this image storage. This buffering will ensure smooth image processing, thereby allowing frames to be retrieved for blending and verification.

[0050] Furthermore, a so-called blending process 105 can be employed to combine enhanced (overlayed) safety-related content from the GPU with real-time, dynamic background images. This blending will ensure that predefined content (e.g., safety-related warning lights) can be seamlessly integrated into the panoramic AR display, thereby providing a coherent and distraction-free visual experience for the vehicle's end users (e.g., drivers).

[0051] The mixed content, such as in the form of a video stream, is then sent via video output path 104 for further suitable (post-)processing, and for the final presentation for display, is represented as 106 (e.g., in the form of a sequence of image frames, a video stream, etc.), which typically represents a transparent AR display seen by the driver. Generally, the display output provides an augmented reality view with key information superimposed on the driver's real-world view, including (but not limited to) speed, navigation data, and especially including an "ESP off" warning icon. Here, any suitable (conventional) (post-)processing techniques may be used depending on the various implementations and / or contexts, so their description can be omitted for brevity and simplicity. For example, as... Figure 1 As illustrated in the example, a suitable module can be configured to inspect warning lights and / or frozen images using (conventional) CRC techniques. Of course, depending on the various implementations and / or contexts, any suitable external device (e.g., coupled via a system-on-chip (SoC)) may also be involved, for example, to perform CRC-based frozen image checks.

[0052] Simultaneously, before being fed to the computer vision (CV) based processing module 107, the display 106 may also be appropriately fed back (e.g., via suitable software- and / or hardware-based techniques, etc.) and may also be (temporarily) stored back into the DRAM 103. The computer vision module 107 can be understood to play a crucial role in verifying the presence (or absence) and correctness of the “ESP Off” warning indicator. In doing so, the computer vision module 107 may be configured to receive at least one image (e.g., one or more image frames as a video stream) representing the visual information 106 displayed on the display. As will be described in more detail below, in general, any suitable computer vision-based techniques can be employed alone or in combination, including (but not limited to) conventional / classical methods, as well as those based on machine learning (ML) or neural networks (NN) or more generally, artificial intelligence (AI). In some possible examples, the computer vision module 107 may also optionally be provided with multiple reference images and / or the expected location of the expected “ESP Off” warning icon to be checked (in... Figure 1 (represented as 108 in Chinese).

[0053] As described above, the output of the computer vision module 107 can then be fed into the same real-time core / processor module 101 or any other suitable (software- or hardware-based) module for final checking / verification to infer the visibility of the predefined content (“ESP off”) within the dynamic display. In some possible examples, particularly if it is determined that the “ESP off” warning light is not correctly visible within the display, the real-time core / processor module 101 or any other suitable module can be configured to generate a signal (e.g., an error signal) 109 indicating that the “ESP off” sign appears invisible in the vehicle display. For example, a (visual or audible) indication signal can be sent to the driver to notify the driver to take any appropriate action accordingly. In some other possible examples, the indication signal can also be sent to any suitable software / hardware module within the vehicle (or even externally), for example, for further troubleshooting.

[0054] Now for reference Figure 2 , Figure 2 Simplified example implementations or working principles of the proposed technology according to some possible example embodiments of this disclosure are illustrated schematically.

[0055] As in Figure 2This example illustratively illustrates how, in order to properly determine the visibility of a predefined piece of content (in this example, again enhanced with real-time ambient imagery) in a (vehicle) display showing dynamic background content (again, enhanced with real-time ambient imagery), an image representing the visual information currently displayed on the display is obtained. As described above, this step can be performed by determining (e.g., by...) Figure 1 The example illustratively illustrates that the "ABS" flag (real-time core / processor) should be presented (overlaid) on the display for notification (e.g., when the vehicle's ABS system is activated, for example, due to emergency braking). Furthermore, the image can be obtained, for example, as an image frame from a video stream representing the information to be presented on the display, as described above. In some other examples (e.g., those not based on a vehicle display), the image may also be obtained in any other suitable manner, such as simply as a screenshot / snapshot.

[0056] Appropriate computer vision-based processing (or algorithms) can then be applied to the image. Also, as mentioned above, in some possible examples, the computer vision-based processing can also receive appropriate information indicating regions within the display expected to present predefined content (sometimes referred to as regions of interest), potentially helping the computer vision-based processing to better check / determine its performance. This is in Figure 2 The area is illustratively represented by a square surrounding the “ABS” logo and an arrow pointing towards the computer vision algorithm. Depending on the implementation and / or context, such information about the region of interest (or region of interest) can be predetermined, pre-configured, or even prescribed (e.g., in standards, etc.). Therefore, the display flow within the area where content should be displayed is analyzed using computer vision-based processing. Of course, as those skilled in the art will understand and recognize, this area / region information is not mandatory, as in some possible examples, the computer vision-based processing can be configured to appropriately scan the entire image area to determine the likely location of the expected dynamic content. For example, in some possible examples, suitable (e.g., classical and / or AI-based) image processing algorithms can be used to dynamically determine the likely location where content (e.g., a warning light) will be displayed. Once the location is determined, the image processing proposed as described in this disclosure can be performed.

[0057] Based on computer vision processing, the result of the inspection can be either that the content is visible or invisible.

[0058] In some possible examples, particularly where the content is not visible, additional checks can be performed, such as comparing the background color with the color of the predefined content. As an illustrative example for understanding, a red warning sign that is (at least partially) superimposed / overlapped with a background object (e.g., a red car) of a similar color may be considered (through computer vision-based processing) inappropriately visible, even if it can be correctly displayed / presented on a monitor. In this case, it is possible to consider (temporarily) changing the color of the warning sign (e.g., changing it to black, etc.) to improve its visibility when presented on a monitor. Of course, as those skilled in the art will understand, any other suitable processing can also be applied here. For example, at least in theory, it is also possible to consider (temporarily) changing the background color in the area (region of interest) in a suitable manner and repeating the processing used to (re)determine the visibility of the predefined content. Furthermore, in some possible examples, at least a warning can be issued to the driver indicating that the content (e.g., an "ABS" warning light) is not visible. Of course, any other suitable (post)processing mechanism can also be invoked, depending on the various implementations and / or contexts. For example, if an incorrect warning sign is found to be displayed, or even if no warning sign is displayed at all, any appropriate troubleshooting procedure can be performed. This is not limited to this disclosure.

[0059] Now for reference Figure 3 , Figure 3 Example flow 300 of a possible implementation of a computer vision-based processing according to some possible exemplary embodiments of the present invention is shown in more detail. This computer vision-based processing can be applied to... Figure 1 or Figure 2 The example system shown. It is worth noting that... Figure 3 This particular example can be understood as representing, in general, one possibility of implementing classical (i.e., non-AI-based) image processing algorithms. Of course, as repeatedly indicated throughout this disclosure, depending on the various implementations and / or contexts, any (or all) of the steps shown below can be replaced (or combined) with any suitable AI-based (e.g., machine learning or neural network) techniques.

[0060] Specifically, in Figure 3In example flow 300, step 301 may involve the generation of shape features (e.g., contour, area, connectivity, etc.), which can be accomplished by any suitable method / algorithm (e.g., edge generation such as Canyon, Sobel, HoughTransform, Wavelets, etc.). In some possible examples, before generating features of dynamic security content from the currently displayed visual information, preprocessing of the region of interest within the display can be performed using appropriate methods (e.g., color conversion operations, binarization operations, morphological operations, etc.). Generally, these features can be understood as useful for identifying the presence of security content within the region of interest in the next step.

[0061] Next, step 302 may involve comparing the generated features, wherein the differences between the features are measured by appropriate methods (e.g., template matching, cross-correlation, etc.) and compared with the differences in the expected security content being inspected, and a metric for these differences is generated based on these differences.

[0062] Subsequently, step 303 can be invoked, which involves techniques based on classification and / or decision matrices. For example, in some possible examples, in this step, machine learning-based techniques (e.g., principal component analysis, support vector machines, etc.) can be applied to the metrics generated from previous steps to classify these metrics and make decisions about the presence or absence of secure content within the region of interest.

[0063] like Figure 3 As illustrated in Example 300, steps 301 to 303 described above can be repeated multiple times (e.g., each time based on the corresponding image frame from the video stream). Therefore, majority voting 304 (or any similar technique) can be performed in the classifier decision regarding those multiple frames in order to increase the confidence level of the final decision.

[0064] As mentioned above, for ease of understanding, Figure 3Example 300 is shown as an illustrative example only and should therefore not be construed as constituting any kind of limitation. As will be understood and appreciated by those skilled in the art, any other suitable computer vision-based techniques may also be employed herein, including (but not limited to) conventional / classical methods, as well as those based on machine learning (ML) or neural networks (NN), or more generally, artificial intelligence (AI) techniques, or even combinations thereof. For example, as another illustrative possible example for the sake of completeness, in some possible implementations, computer vision-based processing may include steps including (but not limited to): applying an edge detection function / algorithm to an area of ​​the display where predefined content will be presented (e.g., the area of ​​the display where predefined content should be presented may be pre-configured / specified, or received as an indication); applying an extraction and reconstruction function / algorithm to the output of the aforementioned function / algorithm; applying a feature extraction function / algorithm to the output of the aforementioned function / algorithm; and finally, determining the visibility of the predefined content on the display based on the output of the aforementioned function / algorithm by using a comparison based on a decision matrix. Of course, it is also possible to repeat the aforementioned function / algorithm for multiple frames.

[0065] at last, Figure 1 The diagram schematically illustrates an example flowchart of a method 400 for determining the visibility (or appearance) of predefined (or pre-determined) content on a display (e.g., a screen, monitor, etc.) showing dynamic background content. This method can be applied to any suitable system setup, which may include, but is certainly not limited to, the above-referenced diagram. Figure 1 or Figure 2 The illustrative description is based on the implementation of the vehicle and warning lights. For example, as mentioned above, such predefined / predetermined content can be in any suitable form, such as warning signs, indicators, warning lights, text, etc. Furthermore, the display can be implemented in any suitable form, such as the display (screen) of a handheld device, computer, vehicle, etc. Therefore, dynamic background content can include any suitable (background) information displayed (presented) on said display.

[0066] Specifically, method 400 may include: at step S410, after determining that predefined content will be overlaid on dynamic background content for presentation on the display, obtaining at least one image representing visual information displayed on the display. Method 400 may also include: at step S420, performing computer vision-based processing based on the obtained at least one image to determine the visibility of the predefined content on the display (or in other words, determining whether the predefined content is visible on the display).

[0067] Thus, this disclosure generally aims to propose an effective, flexible, and reliable mechanism capable of checking / determining the appropriate presence or display of specific (predefined) content (e.g., warning signs) in a display system with a dynamic background (e.g., in a vehicle) (even where the display may be transparent). It is noteworthy that in some possible situations, such as in the automotive industry, ensuring that such content (e.g., warning signs / indicators, warning lights, etc.) is correctly displayed and visible to the driver can be considered critical for safety, regulatory compliance, and user experience. As previously stated, conventional methods (e.g., CRC-based techniques) may be considered insufficient in certain use cases or scenarios (such as dynamic and transparent displays), leading to potential safety risks and regulatory non-compliance.

[0068] It should be noted that the aforementioned device / equipment / system features correspond to corresponding method features; however, for the sake of simplicity, these method features may not be explicitly described. The disclosure of this invention is also intended to extend to such method features. In particular, this disclosure is understood to also relate to methods of manufacturing and / or operating the aforementioned device / equipment / system, and / or to providing and / or arranging the corresponding elements of these devices / equipment / systems.

[0069] It should also be noted that the examples of embodiments of this disclosure can be applied to various applications or system configurations, depending on the underlying technical field. In other words, the examples shown in the foregoing figures that serve as the basis for the examples discussed above (such as power tools) are merely illustrative and do not limit this disclosure in any way. That is, additional existing and proposed new functionalities available in the respective operating environments can be used in conjunction with the examples of embodiments of this disclosure based on the defined principles.

[0070] It should also be noted that the disclosed example embodiments can be implemented in a variety of ways using hardware and / or software configurations. For example, the disclosed embodiments can be implemented using dedicated hardware, dedicated software, and / or hardware associated with software executable thereon. The components and / or elements in the figures are merely examples and do not limit the scope of use or functionality of any hardware, software combined with hardware, firmware, embedded logic components, or combinations of two or more such components that implement a particular embodiment of this disclosure.

[0071] Finally, it should be noted that the description and accompanying drawings merely illustrate the principles of the proposed apparatus / system and method. Those skilled in the art will be able to implement various arrangements, although not explicitly described or shown herein, which embody the principles of the invention and are included within its spirit and scope. Furthermore, all examples and embodiments outlined in this document are primarily intended for illustrative purposes only to aid the reader in understanding the principles of the proposed method. Moreover, all statements herein providing the principles, aspects, and embodiments of the invention, as well as specific examples thereof, are intended to cover their equivalents.

Claims

1. A method for determining the visibility of predefined content on a display showing dynamic background content, the method comprising: After determining that the predefined content will be overlaid on the dynamic background content for display on the screen... Obtain at least one image, the at least one image representing visual information displayed on the display; as well as Computer vision-based processing is performed based on the obtained at least one image to determine the visibility of the predefined content on the display.

2. The method of claim 1, wherein the at least one image is obtained from a video stream as one or more image frames, the video stream representing information to be presented on the display.

3. The method of claim 2, wherein the video stream is generated by mixing frames representing the predefined content with frames representing the dynamic background content.

4. The method of claim 1, wherein the visibility determination output includes information indicating that the predefined content is visible on the display, or that the predefined content is not visible on the display.

5. The method according to claim 1, wherein, If it is determined that the predefined content is not visible on the display, the method further includes: The color of the predefined content is compared with the background color of the area of ​​the display where the predefined content will be displayed.

6. The method according to claim 1, wherein, If it is determined that the predefined content is not visible on the display, the method further includes: A signal is generated that indicates the predefined content is not visible on the display.

7. The method of claim 1, wherein the computer vision-based processing is based on artificial intelligence (AI).

8. The method of claim 1, wherein performing the computer vision-based processing comprises: A shape feature is generated in a region of interest within the display, and the predefined content is presented in the region of interest. The shape feature includes information indicating at least one of the following: contour, area, or connectivity.

9. The method of claim 8, wherein before generating the shape feature, the method further comprises: Receive an indication from the display of the region of interest to display the predefined content.

10. The method of claim 8, wherein the computer vision-based processing further comprises, prior to generating the shape feature: The region of interest is preprocessed, and the preprocessing involves at least one of the following: color conversion operation, binarization operation, or morphological operation.

11. The method of claim 8, wherein performing the computer vision-based processing further comprises: The generated shape features are compared with the shape features of the predefined content to generate one or more metrics that indicate the difference between the generated shape features and the shape features of the predefined content.

12. The method of claim 11, wherein performing the computer vision-based processing based on the obtained at least one image to determine the visibility of the predefined content in the display further comprises: Determining the visibility of the predefined content on the display based on the generated one or more metrics involves techniques based on machine learning and / or decision matrices.

13. The method of claim 1, wherein the computer vision-based processing is repeatedly performed on multiple images; and Determining the visibility of the predefined content on the display also involves: a majority vote on the corresponding outputs of the computer vision-based processing performed on the plurality of images.

14. The method of claim 1, wherein the predefined content is rendered on the display using a graphics processing unit (GPU).

15. The method according to claim 1, wherein the predefined content includes security-related content.

16. The method of claim 1, wherein the predefined content includes text content.

17. The method of claim 1, wherein the display is a vehicle display.

18. The method of claim 1, wherein the display is a display with a dynamic background, and the content is mixed in the dynamic background with transparency.

19. The method of claim 1, wherein displaying the dynamic background content involves: an image of an augmented reality overlay environment.

20. An apparatus comprising a processor and a memory coupled to the processor, wherein the processor is adapted to cause the apparatus to perform the method according to claim 1.

Images